System and method for fetching the latest versions of stored data objects

ABSTRACT

A distributed storage system may store data object instances in persistent storage and may cache keymap information for those data object instances. The system may cache a latest symbolic key entry for some user keys of the data object instances. When a request is made for the latest version of stored data object instances having a specified user key, the latest version may be determined dependent on whether a latest symbolic key entry exists for the specified user key, and keymap information for the latest version may be returned. When storing keymap information, a flag may be set to indicate that a corresponding latest symbolic key entry should be updated. The system may delete a latest symbolic key entry for a particular user key from the cache in response to determining that no other requests involving the keymap information for data object instances having the particular user key are pending.

BACKGROUND

Online and other remote data storage services have become widelyavailable in recent years. In a typical model, a storage service mayprovide storage for backup data, which may be retrieved in the event ofa hardware failure, an accidental deletion of data, or data loss as aresult of a security breach or other malicious act. Storage services mayalso provide long-term remote storage for archival or historicalpurposes.

Although some storage systems support the storing of multiple versionsof a file, they typically do not provide version-aware operations otherthan those used to support fairly simple backup and recovery services.In general, currently available storage services, and their underlyingstorage systems, do not support other use models that may require moreaccess to, and/or control over, multiple versions of a file or otherstored data. For example, a common use case for storage systems thatsupport versioned data is to fetch the latest version of a piece of data(as opposed to fetching an explicitly specified version of the piece ofdata). Determining which of multiple stored versions of a piece of datacan be expensive and time-consuming in a large distributed storagesystem, especially when different versions and/or metadata associatedwith different versions are stored in different storage blocks and/or ondifferent computing nodes.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a flow diagram illustrating a method for storing a data objectin a storage system that supports versioning, according to oneembodiment.

FIG. 2 is a flow diagram illustrating a method for retrieving a dataobject from a storage system that supports versioning, according to oneembodiment.

FIGS. 3A-3F illustrate the effects of various operations on aversioning-enabled bucket in a storage system that supports versioning,according to one embodiment.

FIG. 4 is a flow diagram illustrating a method for efficiently fetchingthe latest version of a stored data object, according to one embodiment.

FIGS. 5A-5C illustrate the ordering of elements in a keymap, accordingto some embodiments.

FIG. 6 is a block diagram illustrating a distributed storage network,according to one embodiment.

FIG. 7 is a block diagram illustrating one embodiment of a keymapsubsystem that may be used to manage keymap information for adistributed storage system.

FIGS. 8A and 8B illustrate the mapping of keys to keymap coordinatorsusing a consistent hashing scheme, according to one embodiment.

FIGS. 9A and 9B illustrate the distribution of different hash valueranges across multiple keymap coordinators, according to one embodiment.

FIG. 10 illustrates the mapping of a particular hash value to the keymapcoordinator on which corresponding keymap information is cached,according to one embodiment.

FIG. 11 is a data flow diagram illustrating the use of an update-nearestparameter, according to some embodiments.

FIG. 12 is a data flow diagram illustrating the behavior of a keymapsubsystem in response to requests to put keymap information that areperformed in a different order than they were issued, according to someembodiments.

FIG. 13 is a flow diagram illustrating a method for a keymap coordinator(KFC) to update keymap information in the keymap subsystem of adistributed storage system, according to some embodiments.

FIG. 14 is a flow diagram illustrating a method for a brick manager toupdate keymap information in a keymap subsystem of a distributed storagesystem, according to some embodiments.

FIG. 15 is a data flow diagram illustrating the behavior of a keymapsubsystem in response to a request to retrieve information identifyingthe latest version of a stored data object, according to someembodiments.

FIG. 16 is a flow diagram illustrating a method for a fetching thelatest version of a data object that is stored in a distributed storagesystem, according to some embodiments.

FIG. 17 is a flow diagram illustrating a method for updating keymapinformation following a DELETE operation in a distributed storagesystem, according to some embodiments.

FIG. 18 is a flow diagram illustrating a method for removing cacheentries that are no longer needed in a keymap subsystem, according tosome embodiments.

FIG. 19 is a flow diagram illustrating a method for clustering keys in adistributed storage system, according to some embodiments.

FIG. 20 is a flow diagram illustrating a method for routing accessrequests for keymap information in a distributed storage system,according to some embodiments.

FIG. 21 is a flow diagram illustrating a method for determining thelatest version of an object in a distributed storage system, accordingto some embodiments.

FIG. 22 is a block diagram illustrating multiplicity in a mapping ofhash values to keymap coordinators, according to some embodiments.

FIG. 23 illustrates one embodiment of a computer system that implementsthe techniques described herein.

While the technology described herein is susceptible to variousmodifications and alternative forms, specific embodiments thereof areshown by way of example in the drawings and will herein be described indetail. It should be understood, however, that the drawings and detaileddescription thereto are not intended to limit the disclosure to theparticular form disclosed, but on the contrary, the intention is tocover all modifications, equivalents and alternatives falling within thespirit and scope of the present disclosure as defined by the appendedclaims.

DETAILED DESCRIPTION OF EMBODIMENTS Introduction

A distributed storage system may store multiple versions of a dataobject (i.e. multiple object instances having the same user key) inpersistent storage, and may cache keymap information for those dataobject instances, e.g., in a keymap subsystem of the distributed storagesystem. Each data object instance may include a user key and a versionidentifier. The keymap information for each data object instance may bestored as a key-value pair and may map the user key of the data objectinstance to a locator and the locator to the instance of the dataobject. In some embodiments, the system may cache a latest symbolic keyentry for at least some of the user keys of the stored data objectinstances, and this latest symbolic key entry may identify the versionidentifier (or version-id) of the latest version of the data objectinstances stored in the distributed storage system and having aparticular user key. For example, the value mapped to the key for aparticular data object version and stored with the key as a key-valuepair in the cache may include an inode for the data object and/or otherkeymap information (once the keymap information has been successfullystored in a brick), and the value mapped to a latest symbolic key andstored with the latest symbolic key as a key-value pair in a latestsymbolic key entry in the cache may be the version-id of the latestversion of the objects having the user key specified by the latestsymbolic key (if the latest version is known).

In some embodiments, if a request is made for the latest version of thestored data object instances having a specified user key (sometimesreferred to herein as a GETNEAREST operation), the latest version may bedetermined dependent on whether a latest symbolic key entry exists forthe specified user key. For example, if a latest symbolic key entryexists for the specified key, the keymap subsystem may return theversion-id identified by the latest symbolic key entry for the specifieduser key. If a latest symbolic key entry does not exist for thespecified key, the keymap subsystem may examine two or more keymapinformation entries that are cached or stored in persistent storage toattempt to determine the latest version of the stored data objectinstances having the specified user key. In some embodiments, whenstoring keymap information for a data object instance having aparticular user key, a flag may be set to indicate that a correspondinglatest symbolic key entry should be created or updated for theparticular user key. In some embodiments, the distributed storage systemmay delete a latest symbolic key entry for a particular user key fromthe cache in response to determining that no other requests for thekeymap information for data object instances having the particular userkey are pending. This GETNEAREST technique is described in more detailbelow, according to various embodiments.

In some embodiments, the distributed storage system may cache keymapinformation for data object instances as key-value pairs in adistributed hash table on two or more computing nodes, where the key isa user key or a composite key that includes a sharable user key (e.g., aname for the object that may be common to multiple data object versionsor instances) and one or more other locally or globally uniqueidentifying elements. The keymap information stored in the distributedhash table for each data object instance may map the key to a locatorand the locator to the instance of the data object. In some embodiments,a request to store keymap information for a data object instance may berouted to a particular one of the computing nodes on which thedistributed hash table is implemented based on a consistent hashingscheme in which a hash function is applied only to a portion of the keyof the data object instance. In this way, related keymap information(e.g., keymap information entries for which a portion of the key iscommon or similar) may be clustered on the same computing nodes. In someembodiments, related keymap information may be stored in the samestorage partition or block (brick) on a single computing node and/or maybe managed by a single keymap coordinator component of a keymapsubsystem.

In some embodiments, a request to retrieve keymap information for a dataobject instance may be routed to a particular one of the computing nodeson which the distributed hash table is implemented based on a consistenthashing scheme in which a hash function is applied only to a portion ofthe key specified in the request. In some embodiments, by clusteringrelated keymap information when it is stored in the distributed hashtable, various operations in the distributed storage system may beperformed more efficiently. For example, the retrieval of keymapinformation for related data objects (such as for multiple versions ofthe same object) may be more efficient when it is clustered on the samecomputing node (e.g., in the same cache) than if a hash function wereapplied to the entire unique (composite) key of each data objectinstance and the resulting distribution of the keymap information acrossthe distributed hash table did not result in such clustering. Theportion of the composite key to which the hash function is applied mayinclude a user key portion of the composite key, a pre-determined numberof bits of the composite key, or be identified using a delimiter in thecomposite key, in different embodiments. This key clustering techniqueis described in more detail below, according to various embodiments.

In various embodiments, the techniques described herein for finding thelatest version of a stored data object and for clustering keymapinformation in a distributed hash table may be implemented independentlyof each other in different types of distributed storage systems,including those that support versioning. In other words, any givenstorage system may implement one of these techniques without the other,or may implement both of the techniques, in different embodiments. Inembodiments in which the storage system implements both the keyclustering techniques described herein and the techniques for findingthe latest version of a stored data object, the GETNEAREST operation maybe more efficient than in storage systems in which keymap informationfor related data object instances is not co-located in a single storagepartition or block (brick) and/or on the same computing node. Forexample, the GETNEAREST operation may only need to examine the cache ofa single keymap coordinator to access a latest symbolic key entry for auser key and/or cached keymap information entries for multiple dataobject instances for the user key in its attempt to determine the latestversion of the data object instances for the user key. Alternatively,the GETNEAREST operation may only need to load a single storage block(brick) from persistent storage to access keymap information entries formultiple data object instances for the user key in its attempt todetermine the latest version of the data object instances for the userkey.

Storage Systems

In various embodiments, the methods described herein may be employed inlocal or remote storage systems, including systems that provide storageservices to users (e.g., subscribers) over the Internet. As noted above,the storage systems described herein may provide support for storingmultiple versions of an object and a variety of use cases that depend onsuch versioning support. In some embodiments, object versioning may beselectively enabled or suspended (e.g., by a data owner or privilegeduser) by toggling the versioning state of a collection of storedobjects. The storage systems described herein may provide a simpleinterface that can be used to store and retrieve object data from asingle stored version of an object or from any of a series of versionsof the object. In various embodiments, multiple value versions of eachobject may be identified by a unique key, e.g., as a key-value pair inwhich the key is a composite key that includes at least a user key. Thesystems and methods described herein may allow users (e.g., storageservice subscribers) and/or client applications (e.g., tools) to store,retrieve and/or delete objects without knowing (or needing to know) theversioning state of the targeted collection of stored objects or (insome cases) the version identifiers of the stored data objects.

In some embodiments, each stored object may include two identifyingcomponents: a user key and a version identifier (or “version-id”). Insuch embodiments, the combination of a user key and a version-id mayuniquely identify an object in a bucket. Objects in the same bucket thathave the same user key but different version-ids may be referred hereinto as versions of one another, object versions of the key, objectinstances, or simply versions of the key.

The following concepts and terms may be used herein to describe storagesystems and methods thereof, according to various embodiments:

-   -   Bucket—A bucket is a logical container in which objects may be        stored in a storage system on behalf of a user. In some        embodiments, every object may be contained in a bucket, and        every object may be addressable using a combination of a bucket        identifier and one or more identifiers of the object itself    -   Object—Objects are the fundamental entities stored in a storage        system. In some embodiments, the stored objects may include        object data and/or metadata. For example, each object may        include a data object portion, and a metadata portion (which may        include default metadata and/or versioning related metadata). In        some embodiments, an object instance may be uniquely identified        within a bucket by the combination of a key (e.g., a        user-supplied name for the object) and a version identifier (or        version-id), which may be separate from the key or may be a        component of composite key that includes the key.    -   Key—A key is an identifier for an object within a bucket. In        some embodiments, every object in a bucket may have exactly one        key, and the combination of a bucket, key, and version        identifier may uniquely identify each object (e.g., each data        object version or instance) stored in the storage system.

In some embodiments, the data object portion of an object may be opaqueto the storage system, i.e. it may be treated as a “black box” entry bythe storage system. In various embodiments, the default metadata of anobject may include, e.g., a name-value pair, the date the object waslast modified, and/or an indicator of the content type (i.e. the datatype of the contents of the data object portion of the object). In someembodiments, the metadata associated with an object may include systeminterjected key-value pairs (containing, for example, a creation dateand/or a last modified date), along with user supplied key-value pairs.

In various embodiments, the storage systems described herein may includesupport for the following storage related tasks: creating and namingbuckets that store data and/or metadata in objects; storing data inbuckets; storing keymap information for data stored in the buckets(where the keymap information maps specific object instances to locatorsassociated with a given user key); retrieving data (e.g., read and/ordownloading the contents of stored objects; retrieving keymapinformation for data stored in the buckets (which may, in turn,facilitate retrieval of the data stored in the buckets); deleting datastored in the bucket; and/or deleting keymap information from persistentstorage and/or a cache.

As noted above, in some embodiments, all objects may be uniquelyidentified in the storage system by a user-key/version-id pair. In suchembodiments, operations that retrieve data from objects, such as variousGET type operations defined by an Application Programming Interface(API), may accept an optional version-id input that identifies aparticular version of an object (i.e. a particular instance of anobject) from which to retrieve data. For each of these APIs, if a userkey is specified, but no version-id is specified, the system may beconfigured to automatically determine the version-id of the latestversion of the object having the specified user key, and to retrievedata from that version of the object (i.e. to automatically fill in thelatest version-id for a specified user key if no version-id isspecified). In some embodiments, operations that create new objects,such as PUT type operations defined by the API, may automaticallygenerate a unique version-id (which may be a unique string) and assignit to the newly created object instance. In some embodiments, aversion-id may be bound to an object instance for the lifetime of theobject and can never be changed. In some embodiments, subsequent to theexecution of a DELETE type operation that specifies a user key, but nota version-id, attempts to retrieve an object instance having thespecified user key without specifying a version-id may return an errorindication. Note, however, that in some embodiments, the storage systemmay not have actually deleted any data objects, or the contents thereof.In some embodiments, in order to permanently delete an object version, aDELETE type request may need to specify both a user key and aversion-id.

Unlike in previous storage systems (e.g., systems in which all objectsare versioned or systems that do not support any object versioning), insome embodiments of the storage systems described herein, users may beable to turn object versioning on and/or off for a given bucket overtime. In such embodiments, various operations performed on a bucketand/or on objects thereof may behave differently depending on whetherversioning has been toggled on (i.e. is “enabled:) or is “off” (or“suspended”) for the bucket. In other words, at least some of theactions taken to perform requested accesses may be dependent on whetherobject versioning is, or has ever been, enabled for the bucket. Forexample, when versioning is toggled on, PUT and DELETE type operationsmay not overwrite an existing object nor actually delete an object, andtoggling versioning off may cause the storage system to stopautomatically creating new versions in response to mutating operationssuch as these. In some embodiments, objects that are PUT into a bucketafter versioning has been enabled cannot be overwritten and cannotoverwrite the existing objects. In some embodiments, objects storedprior to versioning being enabled may be thought of as implicit objectversions that have version-ids with a special sentinel value identifyingthem as implicit object versions.

As noted above, the most common access pattern for a storage system maybe a request to access the latest version of an object (i.e. the latestversion of an object having a specific user key). A naïve implementationof such an access in a system that supports the toggling of versioningstate may require that a symbolic link be generated linking an accessrequest to a user key that does not include a version-id (i.e. anon-versioned access) to a specific version of the object having thespecified user key. In some embodiments, the systems and methodsdescribed herein may provide “latest version” support without the needto explicitly generate such symbolic links, and without relying onlocking data objects and/or versions thereof. The systems describedherein may provide latest version support by an extension of theunderlying data structure in which data and metadata of various objectsare stored and through the use of a GETNEAREST operation defined by theAPI such that a version-id may not need to be specified for all accessesto objects stored in the system.

In some embodiments, the version-ids described herein may includesequencers with the property that the most significant bytes of theversion-id (i.e. a sequencer portion) encode the time at which theversion-id was generated. In one example, the sequencer may encode avalue representing the difference between a predetermined time in thedistance future and the time at which the sequencer (or version-id) wascreated. In some embodiments, the system may store objects that includea series of version-ids (or sequencers thereof) that has a totalordering across all sequencers. In such embodiments, the result of acomparison of the version-ids of the stored objects may be the same asthe result of a comparison of the times at which the version-ids (orsequencers thereof) were created. In some such embodiments, a GETNEARESToperation specifying a given user key, may return either the firstkey-value pair in the total ordering of key-value pairs that includesthe given user key, or the next key-value pair in the total ordering ofkey-value pairs.

In some embodiments, a single Unicode data point (e.g., the nullcharacter or another pre-defined, reserved character) may be introducedinto the version-id as a delimiter character to connect a user key withthe sequencer. In such embodiments, sequences of key-value pairs (forwhich the key may be a composite key consisting of a user key, followedby a connector or delimiter character, followed by a version-id) may bestored within a data structure, e.g., in a keymap, to reflect an overallordering of objects in a particular bucket. Note that in someembodiments, the chosen delimiter character may not be allowed in auser-specified key, or in any user-specified portion of the compositekey described above. However, in some such embodiments, this charactermay be used by the storage system for internal operations. In someembodiments, the version-id for each explicit object version may includea sequencer portion and an ID portion (sometimes referred to herein as alocator), while the version-id for an implicit object version may be aspecial sentinel value. In some embodiments, the ID portion of aversion-id for an explicit object version may be generated by the system(e.g., randomly, or using another suitable approach), and may be uniqueto the target bucket and/or the namespace for the specified user key. Inother embodiments, the ID portion may be assigned by a data owner orprivileged user, and may be required to be unique to the target bucketand/or the namespace for the specified user key. In some embodiments,the ID portion may be a globally unique identifier (GUID). For example,in some embodiments, the composite key for an explicit object versionmay be of the form shown below, and the combination of the sequencer andthe ID portion may be referred to collectively as the version-id for theexplicit object version.

[bucket/user key] [version delimiter] [sequencer] [ID]

In one example, the version delimiter for a composite key may be a nullcharacter (e.g., 0x00), and the version-id may comprise 16 bits (e.g., 8bits for the sequencer portion and 8 bits for the ID portion). Othernumbers and combinations of delimiters (or delimiter bits), sequencers(or sequencer bits), and identifiers (or identifier bits) may beincluded in a composite key, in other embodiments. The use of thecomposite key described above, along with a GETNEAREST operation, may insome embodiments provide a way for a storage system to automaticallyascertain the version-id of (and access) the latest object version for auser key in constant time and without adding any additional indirection.Thus, in some embodiments the number of input/output operations requiredto put, get and/or delete keys in systems that support the toggling ofversioning state and APIs that may or may not include a version-id maynot be significantly different than the number of input/outputoperations used in standard accesses to keys in systems that do notsupport versioning or the toggling of versioning state.

Note that in some embodiments, the efficient logical deletion of anobject may be supported in the underlying data structure of the storagesystems described herein by the inclusion of object versions called“delete marker objects”, or simply “delete markers”. For example, insome situations, a user may wish to block or limit access to some or allversions of a user key without removing the key or its associated datafrom the storage system and/or bucket in which it is stored. In someembodiments, the systems described herein may create delete markerswithin the data structure to denote the logical deletion of the userkey. In such embodiments, the objects having the specified user key maynot actually be removed from the bucket in which they are stored, andmay still be addressable and/or their contents may still be accessible(e.g., to the bucket owner and/or another privileged user). A deletemarker is a special type of object version that may have no dataassociated with it. In some embodiments, a delete marker may be used toindicate that an object having the same user key as the delete markerhas been logically deleted. A delete marker may be created by thestorage system in response to a DELETE type operation that specifiesonly a user key and not a version-id. This newly created delete markermay be the latest version of the user key specified in the DELETEoperation. Note that in some embodiments, multiple delete markers may becreated for a given user key. Since a delete marker marks the logicaldeletion of an object or an object instance, it may be used to supportend-user logical deletion as well as undelete operations, in someembodiments. In some embodiments, the use of delete markers may protectusers from various accidental deletion scenarios.

As noted above, the systems described herein may include operations(e.g., as defined by an API) that support and understand objectversioning, some of which may behave differently depending on thecurrent (and/or past) versioning state of a targeted bucket. Forexample, in some embodiments, an operation for storing a data object inthe system (e.g., a PUT type operation) may guarantee that the objectwill never be overwritten, and that only a privileged user withpermission to delete specific object versions in the bucket in which itis stored (e.g., using a delete type operation that specifies itsversion-id) can delete it. This API may further guarantee that a storetype operation will never overwrite an existing object in a bucket. FIG.1 is a flow diagram illustrating a method for using such an API to storea new data object in a data storage system that supports versioning,according to one embodiment. As illustrated at 110, the method mayinclude initiating a PUT type operation that specifies a user key. Forexample, a requester (e.g. a user, user application, or process) mayissue a PUT OBJECT instruction to a shared storage system or storageservice, and that PUT OBJECT instruction may conform to an API similarto those described herein. The PUT OBJECT instruction may be issued torequest that a particular data object be stored in a bucket that isowned by the requester (e.g., a bucket owned by a user who is a storageservice subscriber), and/or that is currently being accessed. Note thatin some embodiments, if the requester does not have permission to modifythe contents of the bucket, the storage system may return an errorindication in response to an attempt to perform this operation (notshown).

In response to receiving the request (i.e. via the PUT instruction), thestorage system may assign a version identifier (version-id) to the newdata object. In this example, it is assumed that versioning is enabledfor the bucket into which the new data object is to be stored.Therefore, the method may include the storage system generating a new,unique version-id for the new data object, and assigning that version-idto the new data object, as in 120. The storage system may then store thenew data object in the target bucket, along with its assignedversion-id, as in 130. Note that in some embodiments, if versioning isnot enabled for the bucket, the method may include the storage systemassigning a special sentinel version-id value to the new data objectrepresenting an implicit version-id for the non-versioned data object.In that case, if the bucket already stores an existing data objecthaving the same user key as the new data object and that existing dataobject has the sentinel version-id value, storing the new data objectmay include overwriting the existing data object that has the same userkey and the sentinel version-id value. As illustrated at 140 in FIG. 1,the storage system may return a response to the requester indicatingwhether the PUT operation was successful (i.e. whether the data objectwas successfully stored in the target bucket). As illustrated in thisexample, the version-id assigned to the data object in response to thePUT operation may be included in the response returned to requester. Forexample, in some embodiments, the assigned version-id may be included ina header element in the response.

In some embodiments, when an object is stored in a bucket, metadataabout the object may be stored in a data structure (e.g., a keymap)associated with the bucket. This metadata may indicate the user key,version-id value, and a creation/modification date. For example, in somesuch embodiments, when a new object is created, a timestampcorresponding to the date and time at which the new object is createdmay be stored as a creation/modification date for that object in akeymap element associated with the object. If the object is an implicitobject version (e.g., one with the special sentinel version-id valueindicating that it is an implicit object version), thecreation/modification date in the keymap element associated with theobject may be updated when (and if) the implicit object version isoverwritten by a subsequent store operation.

In various embodiments, data objects stored in the system may beretrieved using a GET type operation, e.g., a GET OBJECT operation. Insome embodiments, the requester may need to have permission to accessthe object version being retrieved in order to perform this operation.FIG. 2 is a flow diagram illustrating a method for retrieving a storeddata object from a data storage system that supports versioning,according to one embodiment. As illustrated at 210, the method mayinclude a requester (e.g. a user, user application, or process)initiating a retrieve type operation that specifies a user key. Forexample, the requester may issue a GET OBJECT instruction to a sharedstorage system or storage service, and that GET OBJECT instruction mayconform to an API similar to those described herein. The GET OBJECTinstruction may be issued to request that a data object be retrievedfrom a bucket that is owned by the requester (e.g., a bucket owned by auser who is a storage service subscriber), and/or that is currentlybeing accessed. In response to receiving the request (i.e. via the GETOBJECT instruction), the storage system may return the data objectspecified in the request, as described in more detail below.

As illustrated in this example, if the GET OBJECT instruction does notspecify a version-id, shown as the negative exit from 220, the methodmay include the storage system determining the latest version of thedata object having the specified user key (e.g., using a GETNEARESToperation), as in 230, and designating its version-id as the specifiedversion-id for the GET OBJECT instruction, as in 240. Note that in somecases the data object version that is determined to be the latestversion by the operation illustrated at 230 may not be latest version ofthe object by the time one or more of the subsequent operationsillustrated in FIG. 2 are performed. However, the data object versionthat was determined to be the latest version at 230 may be the target ofthe GET OBJECT operation for the remainder of the GET OBJECT processillustrated in FIG. 2. As illustrated in FIG. 2, the method may includethe storage system returning the stored data object that has thespecified user key and the specified version-id, as in 250. For example,if a version-id was not specified in the GET OBJECT instruction, thestorage system may return the stored data object that has the specifieduser key and the version-id that was determined to be the version-id ofthe latest version of the data object. If the GET OBJECT instructiondoes specify a version-id, shown as the positive exit from 220, thestorage system may return the stored data object that has the specifieduser key and the version-id that was specified in the GET OBJECTinstruction.

Note that in some embodiments, a method for retrieving a stored dataobject from a data storage system that supports versioning may includedetermining whether the specified version is a delete marker (notshown). If so, the storage system may return an error indication to therequester. In some embodiments, the method may include the storagesystem determining whether the requester has permission to access (i.e.to view and/or retrieve) the specified version (not shown). If not, thestorage system may return an error indication to the requester.

The systems described herein may in some embodiments support multipleoperations for listing the contents of a bucket. For example, oneoperation defined by the API may behave in a manner similar to that of acorresponding operation in existing storage systems that do not supportdata object versioning. Such an API may be used to list only theversions of stored data objects that can be retrieved without specifyinga version-id for the data objects (e.g., the latest version of versionedobjects and any implicit versions of non-versioned objects). Anotheroperation defined by the API may be used to list all of the versions ofthe data objects stored in a given bucket, rather than only the versionsof stored data objects that can be retrieved without specifying aversion-id for the data objects. In some embodiments, these listingoperations may benefit from the key clustering techniques describedherein (e.g., by employing them in an operation to find the latestversion of an object when other techniques are not implemented).

As noted above, in some embodiments, a data structure (e.g., a keymap)may store metadata about the objects contained in a storage system or ina bucket thereof. For example, in some embodiments, a keymap for aparticular bucket may include a collection of inodes, each of whichrepresents an object (i.e. a version or instance of an object) stored inthe bucket. Each inode may include metadata associated with the objectit represents, and this metadata may indicate (e.g., directly or throughany of various encoding schemes) its user key, version-id, and creationor modification date. The order of the inodes in a keymap may reflect atotal ordering for the objects in a bucket, e.g., based on the user keysand version-ids of the objects in the bucket. For example, in someembodiments, the inodes may be sorted first by user key (e.g.,lexicographically), and then by version-id. In some such embodiments,the object versions returned by a listing type operation may be orderedfirst in ascending lexicographic order of their keys (e.g., inalphabetical order, A to Z), and then in descending order of theircreation dates (i.e. with the latest version listed first). The use ofkeymaps in determining the latest version of an object is described inmore detail below.

Several examples of keymaps are illustrated in FIGS. 3A-3C, anddescribed in more detail below. FIGS. 3A-3F illustrate the effects ofvarious operations on a bucket in a storage system that supportsversioning, according to one embodiment. As previously noted, in someembodiments, when an object is stored in a versioning-enabled bucket,the old version may not be overwritten. This is illustrated in FIGS. 3Aand 3B. For example, FIG. 3A illustrates a PUT OBJECT operation in whicha new version of an object having the user key “photo.gif” is stored ina versioning-enabled bucket 320 that already contains two objects withthe same name (i.e. with the same user key). In this example, theoriginal object (i.e. the object having the version-id value shown asID=8930287) and another version of the object (i.e. the object havingthe version-id value shown as ID=4857693) remain in bucket 320 followingthe PUT OBJECT operation. In response to the PUT OBJECT operation (whichdoes not specify a version-id), the system generates a new versionidentifier (shown as ID=2121212), and adds the newer version of theobject “photo.gif” (illustrated in FIG. 3A as object instance 310) tobucket 320. The result of this PUT OBJECT operation is illustrated inFIG. 3B, which depicts bucket 320 storing all three of these versions ofthe object “photo.gif”. Note that the functionality described herein forgenerating new version identifiers for objects when they are stored inthe system may prevent users from accidentally overwriting or deletingobjects, and may also provide users the opportunity to retrieve aprevious version of an object.

In some embodiments, any given version of an object (except, e.g., adelete marker object) may be retrieved using a GET OBJECT operation thatspecifies the version-id of the given version. This is illustrated inFIGS. 3C and 3D. In this example, FIG. 3C illustrates a GET OBJECToperation targeting versioning-enabled bucket 320 that specifies userkey “photo.gif” and version-id 8930287. In this example, even though thespecified version of the object is not the latest version of the object,it may be retrieved and returned by the storage system. The result ofthis GET OBJECT operation is illustrated in FIG. 3D, which depicts thatthe data object having user key “photo.gif” and version-id 8930287 isreturned to the requester. Note that in some embodiments, the storagesystem may return a specified object version even if it is not thelatest version of that object, and even if the latest version of thatobject (or any more recent version of that object) is a delete markerobject. Note that, in this example, there is no change in the contentsof bucket 320 as a result of this GET OBJECT operation.

In some embodiments, by default, a GET OBJECT operation may retrieve andreturn the latest version (i.e. the most recently stored version) of anobject having a specified user key, e.g., if no version-id is specifiedfor the operation. This is illustrated in FIGS. 3E and 3F. In thisexample, a GET OBJECT operation targets versioning-enabled bucket 320and specifies a user key “photo.gif” (but not a version-id). Inresponse, the storage system determines the version-id corresponding tothe latest version of the data object having the specified user key, asdescribed in more detail herein. In this example, the storage systemreturns the data object having user key “photo.gif” and version-id2121212 to the requester, since this is the latest version of an objecthaving the user key “photo.gif”. In this example, there is no change inthe contents of bucket 320 as a result of this GET OBJECT operation.Note that in some embodiments, in response to a GET OBJECT operationspecifying a user key but not a version-id, the storage system mayreturn an error indication if the latest version of the object is adelete marker.

As previously note, in some embodiments, the storage system may cachekeymap information for each user key and/or for each object instance(version) stored in the system. In some such embodiments, the cachedkeymap information for each key may include information corresponding tothe value of the keymap information and/or timing informationidentifying the last action associated with the keymap information. Insome embodiments, a cache component of a keymap coordinator may maintaincached keymap information for each brick manger that maintains therespective keymap information. For example, a keymap coordinator mayutilize three brick manager components to maintain the keymapinformation, and the cache component of the keymap coordinator maymaintain cached keymap information for each of the three brick managercomponents. Note that in some embodiments, a keymap coordinator mayupdate the entire cache component (e.g., the entire cache correspondingto all associated brick managers) in the event that it is determinedthat another component has updated a relevant portion of the keymapinformation for at least one user key. In other embodiments, the keymapcoordinator may have the ability to update only a portion of the cachebased on a determination of updated keymap information. In still otherembodiments, the keymap coordinator may be able update only the portionof the cache that corresponds to an individual brick manager.

As previously noted, a common use case for storage systems that supportversioned data is to fetch the latest version of a piece of data, andthis operation is distinct from an API or request to fetch an explicitlyspecified version of the piece of data. Determining which of multiplestored versions of a piece of data can be expensive and time-consumingin a large distributed storage system, especially when differentversions and/or metadata associated with different versions are storedin different storage blocks and/or on different computing nodes. Forexample, an effective cache hit rate for the storage system may benegatively impacted if a GETNEAREST API cannot leverage the cacheappropriately.

In some embodiments, the systems described herein rely upon a consistenthash table to efficiently fetch inode information about a user key. Asnoted above, in systems in which versioning is enabled, all of theversioned data objects (i.e. versioned keys) may be stored using uniquekeys that have the form [bucket/user key] [version delimiter][sequencer] [GUID], where the combination of the sequencer and the GUIIDmay be considered the version-id for the object version (instance). Insome such storage systems, with only this information stored in thecache, in order to determine the latest version of a user key based thesystem may always need to evaluate this information from the persistentsystem (using the brick manager), rather than from the cache, sincecache entries can be evicted.

In order to efficiently support this use case, the storage systemsdescribed herein may include support for the concept of a “latestsymbolic key”. A cache entry for this latest symbolic key may hold theinformation about which version of a user key is the latest version ofthat user key. In some embodiments, a latest symbolic key record may berepresented in a cache in a keymap subsystem (e.g., in a keymapcoordinator) with the composite key [k][V], where k is the user key andV is a special version identifier that cannot be used as a version-idwithin normal cache entries. In one embodiment, the special version-idvalue may be chosen as the Long.MAX_VALUE, which is a value used togenerate other version-id values (e.g., by subtracting the current timefrom Long.MAX_VALUE). Using the notation above, a latest symbolic keymay be of the form [k][Long.MAX_VALUE]. In some embodiments, this latestsymbolic key may be mapped to a value identifying the latest version ofthe objects having the user key k (if the latest version is known), ormay be mapped to a special sentinel value (if the latest version isunknown), and may stored in a latest symbolic key entry the cache in akeymap subsystem along with the value to which it is mapped (e.g., as akey-value pair).

Note that keeping a latest symbolic key entry up-to-date may benon-trivial. For example, there are multiple operations in the storagesystem that can affect which version of an object becomes the latestversion of that object, some of which are described below.

In some embodiments, when a Web server or Web service interface (WS)performs a PUT operation to the keymap coordinator (KFC) for a versionedobject having the user key k. For example, the Web server may perform aPUT operation for an implicit data object instance identified by acomposite key [k][null], or for an explicitly specified data objectinstance identified by the composite key [k][123]. In some embodiments,the Web server may send a special flag “update-nearest=true” to the KFCindicating to the KFC that that the value mapped to the latest symbolickey may need to be updated. In some embodiments, the Web server may onlysends this flag if the user key belongs to a versioning enabled bucket.Note that the inclusion of the flag “update-nearest=true” does notindicate that [k][null] or [k][123] is the latest version, as the Webserver has no way of knowing this information. Instead, this flag merelyindicates that this key composite might represent the latest key, andthat KFC should update the latest symbolic key entry if the newly PUTkey is determined to be the latest version of the key.

In some embodiments, in response to receiving the request, the KFC mayupdate the record represented by [k][Long.MAX_VALUE] with a sentinel,and may perform the PUT to a brick manager (BM). The KFC may also send aflag “calculate-is-nearest=true” to the brick manager.

When BM receives the PUT request with the calculate-is-nearest flag, itmay attempt to calculate whether the version that is being inserted isthe latest version of the user key. Note that versions of a user key mayspan across multiple storage partitions or blocks (bricks). Therefore,in some cases, in order for the brick manager to make this calculation,multiple blocks may need to be loaded, which can dramatically impact PUTperformance. In order to avoid this negative performance impact, in someembodiments, the brick manager may only make a best effort calculation.In such embodiments, the brick manager may attempt to calculate thelatest version only in the context of the block that is currently loaded(e.g., the block in which the PUT is being performed). As a result, itmay sometimes be the case that the brick is not be able to make thiscalculation, e.g., if the key that is being inserted falls on a blockedge. Therefore, there may be three possible outcomes for thiscalculation, indications of which may be returned with the responseheader is-nearest (NEAREST, which indicates that the key that isinserted is the latest version; NOTNEAREST, which indicates that the keythat is inserted is not the latest version; and UNKNOWN, which indicatesthat the brick manager was unable to determined whether the key that isinserted is the latest version).

In some embodiments, on receiving the PUT response, the KFC may examinethe value of the is-nearest header, and if the value is set to“NEAREST”, it may update the record represented by the composite key[k][Long.MAX_VALUE] with the value [k][null] or [k][123], in thisexample. In other cases, the sentinel that was inserted for[k][Long.MAX_VALUE] at the start of the PUT operation may be removed ifthere are no in-flight requests that could potentially affect[k][Long.MAX_VALUE] (i.e. any data or keymap information accessoperations for which the specified user key is k).

In some embodiments, in response to a GETNEAREST call to the KFC for auser key k, the KFC may look for a value of a [k][Long.MAX_VALUE] entryin its cache. In such embodiments, if this value does not exist in thecache, a GETNEAREST call for user key k may be made to the brickmanager. The brick manager may then return the most recent version ofuser key k, e.g., [k][123] in response. In this case, the KFC may updatetwo cache entries. One is a record for the composite key [k][123]. Theother is a record with a composite key [k][Long.MAX_VALUE], that storesthe value [k][123], indicating that the version identified by thecomposite key [k][123] is the latest version of user key k.

A method for efficiently fetching the latest version of a stored dataobject is illustrated in FIG. 4, according to one embodiment. Asillustrated at 410, in this example, the method may include the storagesystem caching keymap entries for each of plurality of objects stored inpersistent storage. In various embodiments, the storage system mayinclude multiple caches (each of which caches a subset of the keymapentries for the storage system) or a distributed hash table, acrosswhich keymap entries are distributed. As illustrated in this example,the storage system may also cache a latest symbolic key entry for atleast one of the keys (e.g., one of the user keys) of the objects storedin persistent storage, as in 420. As described herein, this latestsymbolic key entry may identify the version of the stored object havingthat user key that was determined (at some prior point) to be the latestversion of all the objects stored in the storage system having that userkey.

As illustrated at 430, in this example, the method may include thestorage system receiving a request for the latest version of an object,i.e. the latest version of all the stored objects having a particularuser key that is specified in the request. In response to this request,the storage system may determine the latest version of the object, as in440. As illustrated in this example, determining the latest version ofthe object may be dependent on whether a latest symbolic key entry iscached for the specified user key. Methods for determining the latestversion of an object using such a latest symbolic key entry aredescribed in more detail below. Once the latest version of the objecthas been determined, the method may include the storage system returningthe latest version of the data object having the specified user key, asin 450. For example, once the keymap information for the latest versionof the object has been determined, this information may be used tolocate and access the requested data object from the location at whichit is stored in the storage system.

In various embodiments, individual objects may be identified within thedistributed storage network using various distinct items of information,e.g., a user key, a composite key, a version-id, and/or a locator.Generally speaking, keys, version-ids, and locators may each includealphanumeric strings or other types of symbols that may be interpretedwithin the context of the namespace of the distributed storage networkas a whole, although keys, version-ids, and locators may be interpretedin different ways. In some embodiments, a key (i.e. a user key) may bespecified by a client, or on behalf of a client, at the time acorresponding object is created within a particular bucket (e.g., inresponse to a request by the client to store a new object). If no userkey is specified by the client, a user key and/or a composite key may beassigned to the new object by the distributed storage network. In suchan embodiment, each respective composite key associated with an objectof a particular bucket may be required to be unique within the namespaceof that bucket.

In some embodiments, a locator may represent a globally uniqueidentifier (GUID) of an object among all objects known to thedistributed storage network. That is, while a composite key may beunique to a namespace associated with a particular bucket, a locator maybe unique within a global namespace of all objects within all buckets.For example, a locator may include an alphanumeric string generated bythe distributed storage network to be unique among other locators. Insome embodiments, multiple instances of an object may be replicatedthroughout the physical storage devices used to implement thedistributed storage network, for example to increase data redundancy andfault tolerance. In such embodiments, a unique locator may exist foreach replicated instance of a given object. In some embodiments, keymapinformation may correspond to a mapping of key to one or morecorresponding locators. The keymap information may be maintained by akeymap subsystem and utilized by a distributed storage network to accessobjects maintained by the distributed storage network. In someembodiments, the keymap information may be updated, or otherwisemodified, as instances of specific objects are created, deleted, orotherwise manipulated within the distributed storage network.

FIGS. 5A-5C illustrate examples of the ordering of the elements (e.g.,inodes) in a keymap, according to one embodiment. In these examples,keymap 510 reflects the contents of a bucket that includes objectshaving user keys of A, B, and C. FIG. 5A illustrates the contents ofkeymap 510 after three explicit versions of each of these objects havebeen stored in the bucket (i.e. while versioning was enabled for thebucket). In this example, the version-id values for these objects (eachof which may comprise a sequencer value and a unique identifier, in someembodiments) have been simplified for illustration purposes, and areshown as version-id values of 2, 3, and 4.

As described herein, in some embodiments, the elements in a keymap for agiven bucket may be sorted first by user key (lexicographically), andthen by their version-id values. In embodiments in which the version-idvalues for each explicitly created object version includes a sequencerportion based on the creation date of the object, sorting keymapelements for explicit object versions by their version-id valueseffectively places them in order of the creation dates of those objects(i.e. in reverse chronological order, such that the elements associatedwith the most recently stored object versions appear first in thekeymap). This sorting scheme is illustrated in FIG. 5A, in which thethree versions of an object having user key A are sorted in reverseorder by version-id, and are followed by the three versions of an objecthaving user key B (again sorted in reverse order by version-id), andfinally the three versions of an object having user key C (sorted inreverse order by version-id).

As described herein, a GETNEAREST operation may in some embodiments beinvoked by the storage system to determine the latest version of anobject with a given user key, when no version-id is specified for anoperation specifying the given user key. In some embodiments, thisGETNEAREST operation may search the keymap for the target bucket tolocate the first keymap element (e.g., inode) having the specified userkey. If all of the elements in the keymap having the specified user keyare associated with explicit object versions, and are sorted asdescribed above, the first element in the keymap having the specifieduser key may represent the latest object version with the specified userkey. Thus, in the example illustrated in FIG. 5A, a GET type operationon user key A would return the data of the object represented by thefirst element of the keymap (A#2), which is an object having user key Aand a version-id value of 2, since this is the latest object versionwith user key A. Similarly, a GET type operation on user key B wouldreturn the data of the object represented by the fourth element of thekeymap (B#2), since this is the latest object version with user key B,and a GET type operation on user key C would return the data of theobject represented by the seventh element of the keymap (C#2), sincethis is the latest object version with user key C. Note that in otherembodiments, the elements in the keymap may be sorted using otherschemes, and a GETNEAREST operation may examine two or more keymapentries in order to determine which corresponds to the latest objectversion for a given user key.

FIG. 5B illustrates the contents of keymap 510 after an additionalobject version having user key B is stored in the bucket associated withkeymap 510. Again, it is assumed, in this example, that versioning isenabled for this bucket at the time the additional object version isstored, and that the keymap entries are sorted as described above.Therefore, the newly stored object is an explicit object version withuser key B and a newly generated version-id value. As illustrated inthis example, a new element is added to the keymap between the lastelement representing objects having the user key A (i.e. the elementrepresenting the oldest stored object version with user key A) and thefirst element representing a previously stored version of the objecthaving user key B (i.e. the element representing the most recentlystored version of the object prior to storing this new object version).In this example, the new element, shown in the fourth position in keymap510, reflects a user key of B, and a simplified version-id value of 1,although the version-id value may in some embodiments comprise asequencer value and a unique identifier. In the example illustrated inFIG. 5B, a GET type operation specifying user key B, but not specifyinga version-id, may return the data of the explicit object versionassociated with this new element in keymap 510 (labeled as B#1), becausea GETNEAREST operation specifying user key B may correctly identify thiselement of keymap 510 as the one representing the latest object versionwith user key B. Note that the results of a GET type operation thatspecifies user key A or user key C would be no different than in theexample illustrated in FIG. 5A.

In the examples illustrated in FIGS. 5A and 5B, it is assumed thatversioning is enabled for the bucket associated with keymap 510, andthat no objects having user keys A, B, or C were stored in the bucketprior to versioning being enabled or while versioning was suspended forthe bucket. Therefore, no object versions stored in the bucket havingthese user keys have a version-id value that is a special sentinel valuefor implicit object versions (i.e. a value reflecting that the objectwas stored while versioning was off or suspended for the bucket). In theexample illustrated in FIG. 5C, keymap 510 includes an elementrepresenting one or more objects that were stored in the bucket prior toversioning being enabled or while versioning was suspended. Thiselement, shown as the fourth element in keymap 510, represents animplicit object version having a user key B and a version-id value thatindicates that it is an implicit object version. In this example, theimplicit object version represented by this element in the keymap mayhave been stored (and/or its data overwritten, as described herein) atany arbitrary time during which versioning was off or enabled. However,in various embodiments, the keymap element associated with this implicitobject version may appear first in the ordering of elements associatedwith objects having user key B. For example, in some embodiments, thespecial sentinel value assigned as the version-id for implicit objectversions in the storage system may be a value that is always numericallylower than any other valid version-id in the storage system. In suchembodiments, an element representing an implicit version of an objectwith a given user key may always be the first element in the keymap forobjects with the given user key. In some embodiments, the version-idportion of keymap elements representing implicit object versions may beempty (i.e. it may not contain any value).

In the example illustrated in FIG. 5C, in response to receiving a GETtype operation specifying user key B, but not specifying a version-id,the storage system may invoke a GETNEAREST operation to determine thelatest object version with user key B. However, because the keymapincludes an element representing an implicit object version with userkey B, in this example, it may not be sufficient for the GETNEARESToperation to merely identify the first element in the keymap with userkey B. For example, while an element representing an implicit objectversion for a given user key may always appear first in the keymap(because the elements were sorted by their version-ids), this implicitobject version may not contain the most recently stored data for thegiven user key. Instead, the second element with the given user key(i.e. an element representing the most recently stored explicit objectversion with the given user key) may represent the object version thatcontains the most recently stored data for the given user key.Therefore, in some embodiments, the GETNEAREST operation may need toexamine the creation/modification dates of the element associated withthe implicit object version and an adjacent element associated with anexplicit object version in order to determine which is the latest objectversion for the given user key. In the example illustrated in FIG. 5, aGET type specifying user key B may compare the creation/modificationdates of the implicit object represented by the fourth element of keymap510, shown as “B (implicit)” in FIG. 5C, and the explicit objectrepresented by the fifth element of keymap 510 (labeled as B#1), and mayreturn the data of the object with the most recent creation/modificationdate, as follows:

GETNEAREST (B)=>Latest (B,B#1)

In some embodiments, the storage systems described herein may providethe operations described above using standards-based RepresentationalState Transfer (REST) and/or Simple Object Access Protocol (SOAP)interfaces designed to work with a variety of applications, developmenttools, and/or Internet-development toolkits. These interfaces aresimilar, but there are some differences. For example, in the RESTinterface, metadata is returned in HTTP headers. If the storage systemonly supports HTTP requests of up to a given size (e.g., 4 KB, notincluding the body), the amount of metadata that may be associated witha stored object may be restricted. Using REST, standard HTTP requestsmay be issued to create, fetch, and delete buckets and/or objectsthereof. In various embodiments, a user may employ a toolkit thatsupports HTTP in order to use the REST API, or may use a browser tofetch objects, as long as they are anonymously readable. A REST API mayuse standard HTTP headers and status codes, so that standard browsersand toolkits work as expected. In some embodiments, functionality may beadded to HTTP (for example, headers may be added to support accesscontrol). In such embodiments, the functionality may be added such thatit matches the style of standard HTTP usage. In some embodiments, a SOAPAPI may provide a SOAP interface (e.g., a SOAP 1.1 interface) usingdocument literal encoding. As with the REST interface, users may employa SOAP toolkit to create bindings, and then may write code that usesthese bindings to communicate with the storage system.

In one embodiment, an API that provides access operations in an onlineor remote storage system that supports object versioning may include aPUT type operation that specifies any or all of the followinginformation for the request, some of which may be input by a user, andsome of which may be generated and/or attached to the request by aclient or host process: a user key, a bucket identifier, and/or a dateand/or timestamp reflecting the date/time of the request. In response,the storage system may return any or all of the following: a statusindicator reflecting the success or failure of the operation, theversion-id assigned by the storage system to the data object, a dateand/or timestamp reflecting the date/time at which the data object wasstored (e.g., the date/time at which the operation was completed),and/or an identifier of a server on which the data object was stored.

In this example, the API may include a GET type operation that specifiesany or all of the following information for the request, some of whichmay be input by a user, and some of which may be generated and/orattached to the request by a client or host process: a user key, aversion-id, a bucket identifier, and/or a date or timestamp reflectingthe date/time of the request. In response, the storage system may returnany or all of the following, in addition to the requested object data: astatus indicator reflecting the success or failure of the operation, theversion-id of the data object returned in response to the request, adate and/or timestamp reflecting the date/time at which the data objectwas stored or a date and/or timestamp reflecting the last time thereturned data object was modified, and/or an identifier of a server fromwhich the data object was retrieved. In other embodiments, informationother that than described above may be included in a GET type request orresponse or a PUT type request or response (e.g., a user/subscriberidentifier, an authorization code, a content type, a content size of thedata object (e.g., the number of bytes of content stored in the dataobject), an internal identifier of the object, or an identifier of therequest).

In various embodiments, a storage system the employs the techniquesdescribed herein may store multiple versions of data objects inpersistent storage, may store keymap information for those data objectsin persistent storage, may cache keymap information and/or latestsymbolic key entries for some user key in a distributed hash table. Insome embodiments, the storage system may include a keymap subsystem formanaging keymap information and APIs for storing, updating, andretrieving that keymap information in the keymap subsystem for use inaccessing data object versions (instances) stored in the system. Thestorage system may include multiple computing nodes on which data objectversions (instances) and keymap information are stored and on whichvarious components of the storage system are implemented.

As noted above, various techniques described herein may be employed inlocal or remote storage systems, including systems that provide storageservices to users (e.g., subscribers) over the Internet, and/or storagesystems in a distributed storage network. FIG. 6 illustrates a blockdiagram of a distributed storage network, according to one embodiment.In this example, the distributed storage network 600 includes one ormore storage clients 605. In this example, the storage clients 605 maybe configured to interact with a web services interface 630 via acommunication network 610. Note that in some embodiments, thedistributed storage network may include a load-balancing component (notshown) that is a separate component or that is integrated as asubcomponent of another component of the distributed storage network600.

As illustrated in this example, the web services interface 630 may beconfigured to communicate with a keymap subsystem 620 to obtain specificmappings of keys to object instance locators. One such keymap subsystemis illustrated in FIG. 7 and described in detail below. The Web servicesinterface 630 may also be configured to communicate with a storage nodeinterface 640 to facilitate the sending of instructions to specificnodes of the distributed storage network 600. For example, the storagenode interface 640 may be configured to communicate with a file system,such as via a storage node management (SNM) controller, which may inturn be configured to manage one or more physical storage devices. TheWeb service interface 630 may utilize predefined instructions orcommunications, such as via defined application protocol interfaces(APIs), to communicate with the keymap subsystem 620 and/or storage nodeinterface 640.

In various embodiments, the components illustrated in FIG. 6 may beimplemented directly within computer hardware, as instructions directlyor indirectly executable by computer hardware (e.g., a microprocessor orcomputer system), or as a combination of these techniques. For example,the components of the distributed storage network 600 may be implementedby a distributed system including any number of computing nodes (orsimply, nodes). In various embodiments, the functionality of a givendistributed storage network component may be implemented by a particularnode or distributed across several nodes. In some embodiments, a givennode may implement the functionality of more than one distributedstorage network component.

In the example illustrated in FIG. 6, the storage clients 605 mayencompass any type of client configurable to submit Web servicesrequests to Web services interface 630 via network 610. For example, agiven storage client 605 may include a suitable version of a Webbrowser, or a plugin module or other type of code module configured toexecute as an extension to or within an execution environment providedby a Web browser. Alternatively, a storage client 605 may encompass anapplication such as a database application, media application, officeapplication, or any other application that may make use of persistentstorage resources. In some embodiments, such an application may includesufficient protocol support (e.g., for a suitable version of HypertextTransfer Protocol (HTTP)) for generating and processing Web servicesrequests without necessarily implementing full browser support for alltypes of Web-based data. That is, storage client 605 may be anapplication configured to interact directly with Web services interface630. As described below, storage client 605 may be configured togenerate Web services requests according to a Representational StateTransfer (REST)-style Web services architecture, a document ormessage-based Web services architecture, or another suitable Webservices architecture.

In other embodiments, storage client 605 may be configured to provideaccess to Web services-based storage to other applications in a mannerthat is transparent to those applications. For example, storage client605 may be configured to integrate with an operating system or filesystem to provide storage in accordance with a suitable variant of thestorage model described above. However, the operating system or filesystem may present a different storage interface to applications, suchas a conventional file system hierarchy of files, directories, and/orfolders.

In various embodiments, the communication network 610 may encompass anysuitable combination of networking hardware and protocols necessary toestablish Web-based communications between storage clients 605 and theWeb service interface 630. For example, the communication network 610may generally encompass the various telecommunications networks andservice providers that collectively implement the Internet. Thecommunication network 610 may also include private networks such aslocal area networks (LANs) or wide area networks (WANs) as well aspublic or private wireless networks. For example, both a given storageclient 605 and the Web services interface 630 may be respectivelyprovisioned within enterprises having their own internal networks. Insuch an embodiment, the communication network 610 may include thehardware (e.g., modems, routers, switches, load balancers, proxyservers, etc.) and software (e.g., protocol stacks, accounting software,firewall/security software, etc.) necessary to establish a networkinglink between given storage client 605 and the Internet as well asbetween the Internet and Web services interface 630. It is noted that insome embodiments, storage clients 605 may communicate with Web servicesinterface 630 using a private network rather than the public Internet.For example, storage clients 605 may be provisioned within the sameenterprise as the distributed storage network. In such a case, storageclients 605 may communicate with Web service interface 630 entirelythrough a private communication network (not shown).

FIG. 7 illustrates a block diagram of a keymap subsystem that may beutilized in accordance with the distributed storage network shown inFIG. 6 to manage keymap information, according to one embodiment. Asillustrated in this example, keymap subsystem 710 may include any numberof keymap functionality coordinators 715 (sometimes referred to hereinas keymap coordinators or KFCs). These keymap coordinators may in someembodiments act as a management tier of the keymap subsystem 710, andmay be configured to process requests involving associated keymapinformation. As described in greater detail herein, the keymap subsystem710 may communicate with one or more hash router components 700, whichmay be configured to select, or otherwise identify, which keymapcoordinator 715 should process requests for keymap information for anidentified key.

As illustrated in this example, a hash router component 700 may includean interface component 705 for obtaining consistent hashing informationthat facilitates the identification of the keymap coordinators 715 thathave been assigned to process requests for specific keymap information.As illustrated in FIG. 7, hash router component 700 may be implementedas a stand-alone component separate from the keymap subsystem 710, insome embodiments. In other embodiments, hash router component 700 may beimplemented as part of another component with the distributed storagenetwork 600, such as the web service interface component 630 shown inFIG. 6, or another component with the distributed storage network 600.One or more hash router components 700 implemented within thedistributed storage network 600 may be accessible by other componentsvia various interfaces, such as application protocol interfaces (APIs).

As illustrated in this example, each of the keymap coordinators 720 mayinclude (or otherwise be associated with) a keymap information cachecomponent 720 for caching keymap information, as described in greaterdetail below. In various embodiments, the keymap information cachecomponent 720 may be implemented in one or more hardware components,such as internal memory, external memory, shared memory, etc.Accordingly, a keymap cache component 720 may represent a logical cachemaintained by a single keymap coordinator 715, or may be shared betweenmultiple keymap coordinators 715. As described herein, a keymap cachecomponent 720 may be logically considered as having separate cachecomponents corresponding to various other components (such as respectivekeymap coordinators) regardless of whether the keymap cache component isimplemented as a single cache or as multiple caches. Accordingly, thelogical implementation of each keymap cache component 720 may not needto match the physical implementation of such a keymap cache componentwithin a keymap coordinator 715.

As illustrated in FIG. 7, the keymap subsystem 710 may include aplurality of brick manager components 730 for managing, or otherwisecontrolling, the storage of keymap information in one or more defined astorage data structures, sometimes referred to herein as “bricks.” Oneor more bricks may be associated with partitions or storage blocks onphysical storage devices. The allocation and communication withindividual bricks to manage the storage of the keymap information may becontrolled through the brick managers 730. In some embodiments, thebrick managers 730 may receive input/commands regarding keymapinformation from other components in the distributed storage network600. In some cases, the inputs/commands received from the othercomponents (such as reconciler agent daemons or anti-entropy agents,replication agents, repair agents, diagnostic tools, and the like) maybe independent of the operations of keymap coordinators 715. In someembodiments, specific brick managers 730 may receive inputs/commandsfrom multiple keymap coordinators 715. As described in greater detailbelow, in some embodiments, multiple keymap coordinators and othercomponents in the distributed storage system may have the ability andauthority to access and modify the keymap information. In someembodiments, the brick manager components 730 may function as the sourceof keymap information that is cached by the keymap coordinators 715.

In some embodiments, a keymap information request may be processed bythe keymap subsystem 710 illustrated in FIG. 7 as in the followingexample. In this example, a hash router component 700 may receive akeymap information retrieval request, and may direct it toward thespecific keymap coordinator 715 associated with an identified key. Inthis example, the hash router component 700 may utilize a consistenthash scheme for hashing keys to generate a unique hash for each key. Thepossible hash values, or known hash values, may be subsequentlyallocated to each of the keymap coordinators 715 in the keymap subsystem710 via the interface component 705. Accordingly, using the consistenthashing scheme, the hash router component 700 may identify which keymapcoordinator 715 should be able to process the received keymapinformation retrieval request. Note that typical hashing algorithms maystore key-value pairs in a hash table according to values generated byapplying a hash function to the key. However, as discussed in detailbelow, in some embodiments, the hash router may use only a portion ofthe key, and this may allow related key-value pairs to be co-located.

FIGS. 8A and 8B illustrate the mapping of keys (e.g., user keys) tokeymap coordinators, according to one such hashing scheme. In thisexample, a distributed storage system includes a number of storagebuckets 802, each of which stores objects having one or more user keys.For example, one bucket stores objects having user keys of E4 (shown inFIG. 8A as 804) and E2 (806). Another bucket stores objects having userkey E1 (808). A third bucket stores objects having user keys of E3(810), E7 (812), and E8 (814). Yet another bucket stores objects havinguser key E6 (816). As illustrated in FIG. 8B, the keymap information foreach of these user keys may be distributed across a fleet of keymapcoordinators (shown in FIG. 8B as KFC fleet 820) according to aconsistent hashing scheme. In other words, a hash function h is used tomap user keys from the storage system domain into the keymap subsystemdomain.

As illustrated in this example, the keymap information for various userkeys associated with a given storage bucket may be mapped (e.g.,physically and/or logically) to different keymap coordinators. Forexample, a first keymap coordinator (shown as KFC1) may store keymapinformation for user key E4; another keymap coordinator (KFC2) may storekeymap information for user keys E7 and E2; a third keymap coordinator(KFC3) may store keymap information for user keys E1 and E8; and afourth keymap coordinator (KFC5) may store keymap information for userkeys E6 and E3. Note that in other embodiments, there may be aone-to-one mapping between the user keys associated with a given storagebucket and a respective keymap coordinator (physically or logically). Inthis example, when a request for keymap information is sent from a webserver of a web server fleet 818 to a KFC fleet 820, the particular keyfor which keymap information is requested may be hashed using the samehash function h in order to route the request to the appropriate keymapcoordinator in KFC fleet 820.

The selection of keymap coordinators from a keymap subsystem utilizingthe consistent hashing of keymap information (according to someembodiments) may be further illustrated by the example shown in FIGS. 9Aand 9B. In this example, various keymap coordinators (which may besimilar to the keymap coordinators 715 illustrated in FIG. 7 anddescribed above) may be selected for the storage of keymap informationand/or for processing keymap information requests. As previouslydescribed, a hash router component (e.g., one similar to hash routingcomponent 700 of FIG. 7 and described above) may utilize a consistenthash scheme for hashing keys to generate a unique hash for each key. Inthis example, the set of potential, or known hash values, may berepresented as a ring 900. As illustrated in FIG. 9, this ring 900 ofpotential hash values may be further broken down into a set of hashvalue ranges 902, 904, 906, 908, 910, 912, 914, and 916. As shown inthis example, the ranges defined for each keymap coordinator may be ofdifferent lengths. Alternatively, the hash value ranges may be of equalsize.

As illustrated by table 920 in FIG. 9B, each of the hash value ranges inthe ring 900 may be mapped to one of the keymap coordinators in thekeymap subsystem. In one embodiment, each keymap coordinator may beassociated with a single range with the ring 900. In other embodiments,such as that illustrated in FIGS. 9A and 9B, some or all of the keymapcoordinators in a keymap subsystem may be associated with multipleranges of hash values. In some such embodiments, each keymap coordinatormay be associated with the same number of hash value ranges. Table 920illustrates the assignment of ranges 902-916 in the ring 900 to keymapcoordinators “KFC1,” “KFC2,” “KFC3” and “KFC4” in a keymap subsystem. Asillustrated in columns 922 and 924 of the table 920, each keymapcoordinator is associated with more than one range from the ring 900, inthis example. Note that the assignment of ranges with the ring does notnecessarily have to follow a consistent order or pattern. For example,in various embodiments, the order of the assignment of hash value rangesto keymap coordinators may be random, or may be determined in accordancewith selection/processing criteria other than, or in addition to, aconsistent hashing scheme.

As previously described, hash router components may be associated withor found in various components within a distributed storage network. Forexample, a Web services interface may include a hash router componentfor directing requests to an appropriate keymap coordinator. In such anembodiment, individual copies of a mapping table, such as table 920, maybe maintained at each hash router component via its interface component.The information associated with the definition of hash ring ranges andtheir assignments to keymap coordinators, as well as any otherinformation associated with the selection of keymap coordinators, may begenerally referred to as “hash routing information.” In someembodiments, various hash router components may utilize variousprotocols to update or propagate the hash ring range information. Forexample, the distributed storage network may utilize a peer-to-peerdistribution communication protocol, such as a gossip (or epidemic)computer to computer communication protocol, to allow various componentswithin the distributed storage network to maintain updated (and,eventually, consistent) hash ring range information among the hashrouter components. In such an embodiment, each hash router component maytransmit and receive the hash routing information from other, known hashrouter components, but each hash router component may not need to knowabout all possible hash router components in the distributed storagenetwork. In some embodiments, each hash router component may processinformation from multiple hash router components to determine the mostcurrent version of the hash routing information. Accordingly, in suchembodiments, the selection of an appropriate keymap coordinator may beimplemented as a non-centralized process in which the hash routinginformation is updated within the distributed storage network.

As previously described, in some embodiments, the keymap coordinatorsmay implement a cache component for maintaining keymap information. Thecache component may be a proxy cache that maintains keymap information(e.g., keymap values) for each of the brick managers associated with, orotherwise in communication with, the corresponding keymap coordinator.In such embodiments, a keymap coordinator may be able to utilize cachedkeymap information to response to a keymap retrieval request. In someembodiments, the keymap coordinators may utilize memory optimizationtechniques for maintaining keymap information for a plurality of brickmanager components. In some embodiments, the keymap coordinators mayutilize generation identifier information to manage the keymapinformation maintained in their cache components.

FIG. 10 illustrates the mapping of a particular hash value (i.e. a hashvalue generated from a particular key or a portion thereof) to thekeymap coordinator on which corresponding keymap information is cachedand/or otherwise stored, according to one embodiment. In this example,ring 1000 represents a ring of hash values that may be generated fromkeys in the storage system, and these values range from 0 to (2³²−1).Each point on ring 1000 corresponds to a hash value that may begenerated from a key according to a hash function h. In this example,the hash values contain 32 bits, and point 1002 represents a hash valueof zero (or, alternatively, 2³²).

In this example, the points labeled KFC1 (1012), KFC2 (1004), KFC3(1014), KFC4 (1006), KFC5 (1008), and KFC6 (1010) represent the lasthash value in a range corresponding to the named keymap coordinator. Forexample, point 1010 corresponds to the last hash value in the range ofhash values cached by keymap coordinator KFC6, according the hashfunction: h(KFC6.id). In this example, all of the hash values on ring1000 in the range of values between this value and the hash valuerepresented by point 1012 (shown in bold on ring 1000 while movingclockwise) are cached by KFC1. Similarly, the hash values in the rangeof hash values between those of points 1004 and 1006 (shown by thedashed line in FIG. 10) are cached by KFC4; the hash values in the rangeof hash values between those of points 1008 and 1010 (shown by dottedline in FIG. 10) are cached by KFC6; and so on.

In this example, if the hash value generated from a given keycorresponds to point 1016 on ring 1000, the keymap coordinator on whichkeymap information for this key should be stored and/or from which itshould be retrieved may be found by moving clockwise on ring 1000 frompoint 1016 to the point corresponding to the next keymap coordinatorboundary (in this case, point 1012). This point corresponds to KFC1, andidentifies KFC1 as the keymap coordinator for keymap information for thegiven key.

In embodiments that employ latest symbolic key entries, when a webservices interface issues a PUT request to the keymap subsystem, anadditional parameter may be added to the PUT operation to support thismechanism. For example, a PUT request may include an “update-nearest”parameter (which may indicate that a cache entry for the latest symbolickey may need to be updated). More specifically, this parameter mayindicate to the KFC that the PUT operation may affect the value mappedto the latest symbolic key for a given user key and that the KFC shouldattempt to either update the latest symbolic key entry for the givenuser key in its cache or invalidate it from the cache. This parametermay be Boolean in type, in various embodiments. In some embodiments, theupdate-nearest parameter may be implemented as a flag to be included inrequests and responses only if it is “true”, or as a parameter that isalways included in the appropriate request and response messages andthat has one of two or more values (including “true” and “false”).

FIG. 11 is a data flow diagram illustrating the use of this additionalparameter, according to some embodiments. As illustrated in thisexample, a web service interface of a distributed storage system (shownas WS 1110) may issue a request to the keymap subsystem to store keymapinformation for a PUT type operation. For example, a request to PUT adata object may be received from a user (e.g., a storage servicesubscriber) or a requesting application, in various embodiments, and WS1110 may issue a request to store keymap information for that PUToperation. This request (shown as 1115) may include a user key for thedata object to be put, a version-id for the data object to be put, and avalue for the Boolean parameter update-nearest. As described herein, therequest may be routed to an appropriate keymap coordinator (shown as KFC1130) according to a consistent hashing scheme applied to keys (orportions thereof). The KFC 1130 may pass a similar request to anappropriate brick manager (shown as BM 1150), and this request (shown as1125) may include the user key, the version-id, and a value for theBoolean parameter “calculate-is-nearest”. If the KFC sees that the valueof the flag “update-nearest” is true, it may set the value of theparameter “calculate-is-nearest” to true in request 1125.

As described above, BM 1150 may store the keymap information for the PUToperation in a particular storage partition or block (brick), in someembodiments. As illustrated in this example, if thiscalculate-is-nearest flag is set to true, the brick manager (BM 1150)may also attempt to calculate whether the data object being put will bethe latest version of the objects stored in the storage system havingthe specified user key (i.e. whether it corresponds to the “nearest”version of the user key within the context of the block in which keymapinformation for objects having the specified user key are being stored).BM 1150 may return a response to request 1125 that includes theversion-id value of the data object being put, and a value for an“is-nearest” parameter. This response is shown as 1145 in FIG. 11. If BM1150 is able to determine that the version-id corresponds to the“nearest” version of the specified user key without performing anyadditional block loads, the response 1145 may include an “is-nearest”flag that is set to “NEAREST”. If BM 1150 is able to determine that theversion-id does not correspond to the “nearest” version of the specifieduser key, the response 1145 may include an “is-nearest” flag that is setto “NOTNEAREST”. If BM 1150 cannot determine whether the version-idcorresponds to the “nearest” version of the specified user key, theresponse 1145 may include an “is-nearest” flag that is set to “UNKNOWN”.

As illustrated in this example, if response 1145 indicates that theversion-id corresponds to the “nearest” version of the specified userkey, KFC 1130 may update the latest symbolic key entry updated for thisuser key to indicate that the version-id included in response 1145corresponds to the latest version of the specified user key. Forexample, it may replace a value currently mapped to the latest symbolickey in a latest symbolic key entry (e.g., a special sentinel value oranother version-id value that was previously mapped to the latestsymbolic key) with this version-id. If response 1145 does not indicatethat the version-id corresponds to the “nearest” version of thespecified user key (i.e. if the response includes an is-nearest flagvalue of NOTNEAREST or UNKNOWN), KFC 1130 may remove or invalidate thelatest symbolic key entry for the specified user key in the KFC cache.As illustrated in FIG. 11, KFC 1130 may return an indication of thestatus of this operation (i.e. the operation to store keymap informationfor a newly PUT object) to the Web service interface (1110). Forexample, the response may include an indication that the operation tostore keymap information was successful or (if it was not successful)may include an error indication.

In some embodiments, the storage system may employ various mechanisms toassist in achieving eventual consistency in the system, e.g., whenupdates to keymap information are performed in a different order thanthe order in which they were issued. For example, in some embodiments,timestamp values (e.g., the sequencers described above) may be includedin keymap access requests and responses to prevent errors due toout-of-order processing of requests. FIG. 12 is a data flow diagramillustrating the behavior of a keymap subsystem in response to requeststo put keymap information that are performed in a different order thanthey were issued, according to some embodiments. In this example, twosuch requests are issued from a Web services interface (WS 1210). Theserequests (shown as 1215 and 1220) correspond to two different putoperations for the same user key (k). The requests include timestampvalues (t₁ and t₂) indicating, e.g., the time at which each request wasissued, or the time at which the corresponding version-id values (v₁ andv₂) for the put operations were generated. In this example, request 1215was issued prior to request 1220. As described herein, the requests maybe routed to the appropriate keymap coordinator (KFC 1230) in the keymapsubsystem for the specified key according to a consistent hashingscheme, and KFC 1230 may pass the requests on to an appropriate brickmanager (BM 1250) as requests 1225 and 1235, respectively.

In this example, request 1225 is passed to BM 1250 prior to request1235. However, the response to request 1235 (shown as 1245) is returnedprior to the response to request 1225 (shown as 1255). If timestamps hadnot been included in each of the original requests, KFC 1230 may (afterresponse 1255) update a latest symbolic key entry in its cacheincorrectly. For example, when BM 1250 stores keymap information forversion v₂ of the user key to a brick and returns response 1245 to KFC1230, KFC 1230 may update the latest symbolic key entry for k toindicate that v₂ is the latest version of the stored objects that havethe user key k. Later, if BM 1250 stores keymap information for versionv₁ of the user key to a brick and returns response 1255 to KFC 1230, KFC1230 might have updated the latest symbolic key entry for k to indicatethat v₁ is the latest version of the stored objects that have the userkey k. However, this would have been incorrect, as the PUT operationcorresponding to version-id v₂ was issued more recently. In someembodiments, the keymap subsystems described herein may employ timestampvalues and/or other mechanisms to prevent such incorrect updates tolatest symbolic key entries. For example, if the timestamp values t₁ andt₂ are included in responses 1245 and 1255, respectively, the KFC mayexamine those timestamp values and compare them to a sequencer valuestored in the latest symbolic key entry (which, as described above, maycorrespond to and/or have been generated from a timestamp value in aprevious operation) before deciding whether to update (or eveninvalidate) the latest symbolic key entry. For example, when response1255 is returned and includes an earlier timestamp (t₁) than a timestampcorresponding to the sequencer portion of the version-id stored in thelatest symbolic key entry after the receipt of response 1245 (whichincluded timestamp t₂), the KFC may remove or invalidate this cacheentry, rather than incorrectly updating it with the information fromresponse 1255. In other words, the KFC may not update the latestsymbolic key entry for a given user key (but may remove or invalidateit) if it receives a response from the brick manager that includes atimestamp/sequencer value higher (older) than one already stored in thelatest symbolic key entry for that user key.

In some embodiments, the storage system may provide another mechanismthat may be employed in a keymap storage system to assist the system inachieving eventual consistency. This mechanism employs a specialsentinel value (e.g., a different sentinel value than the sentinel valuefor an implicit object version) in one or more KFC cache entries(including a latest symbolic key entry) to indicate that an update ofthe keymap information and/or an update of a latest symbolic key entryis in progress for a given user key.

One embodiment of a method for a keymap coordinator (KFC) to updatekeymap information in a keymap subsystem of a distributed storage systemis illustrated by the flow diagram in FIG. 13. As illustrated at 1310,in this example, the method may include the KFC receiving a request froma web services interface (WS) to store keymap information for a PUT typeoperation, and the request may specify a particular key (which may be acomposite key that includes at least a user key). For example, the webservices interface may send this request to the KFC in response to areceiving a request from a user (e.g., a storage service subscriber) ora requesting application to PUT a data object in a distributed storagesystem that supports versioning. As illustrated in this example, anddescribed above, the request may include an update-nearest parameterwhose value indicates whether there is a possibility that the dataobject being put will be the latest version of the objects stored in thedistributed storage system having the specified user key. In suchembodiments, the web services interface may set the value of thisparameter to true if the specified key belongs to a versioned object oran object in a versioned bucket in the storage system. In otherembodiments, the web services interface may include an update-nearestflag in the request (or send an update-nearest flag along with therequest) if the specified user key belongs to a versioned object or anobject in a versioned bucket in the storage system. If the specifieduser key does not belong to a versioned object or bucket, the webservices interface may set the value of the parameter update-nearestparameter to false (or may not include an update-nearest flag in, oralong with the request).

As illustrated in FIG. 13, the method may include the KFC writing aspecial sentinel value in the latest symbolic key entry for thespecified user key indicating that the “nearest” version-id value forthis user key may change, as in 1320. In other words, the KFC mayreplace the value mapped to the latest symbolic key in the latestsymbolic key entry with this special sentinel value. Note thatoverwriting the value for the latest symbolic key entry may be used toeffectively invalidate the latest symbolic key entry, on the assumptionthat the PUT operation (or another operation in progress) will affectthe latest symbolic key entry even if the update-nearest flag is notset. For example, other operations in progress may affect the latestsymbolic key entry, and the absence of this flag (or a correspondingparameter value of “false”) may merely indicate that it is unknownwhether the PUT operation will affect the latest symbolic key entry. Asillustrated in this example, the method may also include the KFC adding(or updating) a cache entry for this PUT request and setting the valuethat is mapped to the particular key specified for the PUT operation inthis cache entry to a special sentinel value (e.g., the same sentinelvalue or a different sentinel value, in various embodiments) to indicatethat an operation is in progress.

As illustrated in this example, if the update-nearest parameter value isfalse (or no update-nearest flag is sent with the request), shown as thenegative exit from 1330, the method may include the KFC sending therequest to a brick manager, and receiving a response from the brickmanager indicating that the keymap information for the PUT operation hasbeen stored in a brick, as in 1335. Note that in this case, although thelatest version may change, no attempt will be made to calculate thelatest symbolic key entry for the specified user key.

If the update-nearest parameter value is true (or if an update-nearestflag is sent with the request), shown as the positive exit from 1330,the method may also include the KFC sending the request to a brickmanager, and the request may include a calculate-is-nearest flag or acalculate-is-nearest parameter value set to “true”, as in 1340. Asillustrated at 1350, the KFC may receive a response from the brickmanager indicating that the keymap information for the PUT operation hasbeen stored in a brick, and the response header may include a value ofan is-nearest parameter. If the is-nearest parameter is set to NEAREST,indicating that the data object being PUT will be the latest version ofthe objects stored in the distributed storage system having thespecified user key, the response may also include an identifier of theversion-id of this latest version.

As illustrated in FIG. 13, after receiving a response from the brickmanager, the KFC may update the cache entry corresponding to this PUT toreplace the special sentinel value in the entry with some or all of thekeymap information (e.g., an inode) for the data object being PUT, as in1360. In addition, if the update-nearest parameter value is true (or ifan update-nearest flag was sent with the request) and the value of theis-nearest parameter is NEAREST, shown as the positive exit from 1365,the KFC may update the latest symbolic key entry for the specified userkey, i.e. the KFC may replace the sentinel value mapped to the latestsymbolic key with the version-id value returned in the response from thebrick manager, as in 1370. If the update-nearest parameter value isfalse (or if no update-nearest flag was sent with the request) and/or ifthe value of the is-nearest parameter is not NEAREST (e.g., if it isNOTNEAREST or UNKNOWN), shown as the negative exit from 1365, the methodmay include the KFC removing (or invalidating) the latest symbolic keyentry for the specified user key in its cache (assuming there are norelated requests pending), as in 1380. If there are pending requestsspecifying this user key, the sentinel value may be maintained in thelatest symbolic key entry for this user key until all of the pendingrequests have been resolved, as described in more detail below.

One embodiment of a method for a brick manager to update keymapinformation in a keymap subsystem of a distributed storage system isillustrated by the flow diagram in FIG. 14. As illustrated at 1410, inthis example, the method may include the brick manager receiving arequest from a keymap coordinator (KFC) to store keymap information fora PUT operation, and this request may specify a particular key (whichmay include at least a user key). As described above, the request mayinclude a calculate-is-nearest flag or parameter value whose value (orpresence) is determined by the KFC. As illustrated in this example, themethod may include the brick manager loading a particular storagepartition or block (brick) in which the keymap information is to bestored, and storing the keymap information for this PUT operation inpersistent storage in the particular storage partition or block (brick),as in 1420. If the calculate-is-nearest flag is not included in therequest (or the value of such a parameter is false), shown as thenegative exit from 1430, the method may include the brick managerreturning a response to the KFC indicating that the keymap informationhas been stored, as in 1435.

If the calculate-is-nearest flag is included in the request (or thevalue of such a parameter is true), shown as the positive exit from1420, the method may include the brick manager attempting to determinewhether the version-id for this PUT operation will be the latestversion-id for the objects having the specified user key. If a besteffort calculation of the latest version is not possible, shown as thenegative exit from 1450, the brick manager may return a response to theKFC indicating that the keymap information has been stored, and theresponse may include an is-nearest parameter value of UNKNOWN, as in1455. Note that in some embodiments a parameter value of UNKNOWN may bereturned if the keymap information needed to determine the latest objectversion for the user key crosses a block boundary, for example, andcannot be easily or efficiently accessed by the brick manager. Note thatthe is-nearest parameter value may be included in the response header,in some embodiments.

If a best effort calculation of the latest version is possible, shown asthe positive exit from 1450, and the brick manager determines that theversion-id will be the latest version of the object with the specifieduser key (shown as the positive exit from 1460), the brick manager mayreturn a response to the KFC indicating that the keymap information hasbeen stored, and the response may include an is-nearest parameter valueof NEAREST, as in 1470. If a best effort calculation of the latestversion is possible, shown as the positive exit from 1450, and the brickmanager determines that the version-id will not be the latest version ofthe object with the specified user key (shown as the negative exit from1460), the brick manager may return a response to the KFC indicatingthat the keymap information has been stored, and the response mayinclude an is-nearest parameter value of NOTNEAREST, as in 1480.

Various APIs supported in a distributed storage system may be modifiedto employ (or take advantage of) a latest symbolic key entry, such asthat described above. For example, the code for a GET type operation ata keymap coordinator (e.g., a KFC GET API) may accept a new parameter,“update-nearest”, which indicates that the latest symbolic key entry fora user key specified in a GET type operation may need to be updated. Insome embodiments, this parameter may not be used for the GET interactionitself, but may used when a repair interaction is initiated in the KFC.For example, a repair may trigger a PUT operation, and this parametermay be used to update the latest symbolic key entry in the KFC cache, asdiscussed herein.

In some embodiments, a new API, “GETNEAREST”, may be supported by keymapsubsystem (e.g., by the keymap coordinators and brick managers). Theparameters and the response of the GETNEAREST API may be simple. Forexample, GETNEAREST may require only on input parameter (a user key) andmay include other optional parameters. In some embodiments, thesemantics for these parameters may be same as in the GET call describedabove. In some embodiments, the response of GETNEAREST may also besimilar to that of the GET call, albeit with different response verbs(e.g., GETNEARESTDATA and GETNEARESTNOKEY).

FIG. 15 is a data flow diagram illustrating the behavior of a keymapsubsystem in response to a request to retrieve information identifyingthe latest version of the objects stored in a distributed storage systemand having a specified user key, according to some embodiments. Asillustrated in this example, a GETNEAREST request 1515 may be sent froma web services interface (WS 1510) to a keymap subsystem, which mayroute it to an appropriate keymap coordinator (KFC 1530), and therequest may include a particular user key (k). For example, theGETNEAREST request may be initiated directly by a user (e.g., a storagesystem subscriber) or requesting application through the web servicesinterface, or may be initiated by the web services interface in responseto a request that depends on this information, in various embodiments.In some embodiments, a hash router component of the keymap subsystem maydetermine an appropriate KFC to process the request using a consistenthashing scheme based on the key (or a portion thereof), as describedherein.

As illustrated in this example, KFC 1530 may route the GETNEARESTrequest to an appropriate brick manager (BM 1550) as request 1525. Brickmanager 1550 may determine the version-id of the latest version of theobjects stored in the distributed storage system having the specifieduser key, using any of the techniques described herein or any othersuitable technique, and may return a response 1535 to KFC 1530 thatincludes the user key and the version-id for the latest version of theobjects with that user key. In some embodiments, KFC 1530 may create orupdate one or more entries in its cache (e.g., a latest symbolic keyentry for this user key, and/or a cache entry for the particular dataobject version) to reflect this information, and/or may return aresponse 1555 to WS 1510 including this information. The web servicesinterface may use this information to access the data object instanceassociated with the information and return it to the requester in aGETNEARESTDATA response (not shown). In some embodiments, if no objectsare stored in the distributed storage having the specified user key, aresponse of GETNEARESTNOKEY may be returned from brick manager 1550 toKFC 1530, from KFC 1530 to WS 1510, and from WS 1510 to the requester.

One embodiment of a method for a fetching the latest version of a dataobject stored in a distributed storage system is illustrated by the flowdiagram in FIG. 16. As illustrated at 1610, in this example, the methodmay include a requester initiating a GET type operation specifying auser key, but not a version-id. For example, in some embodiments a user(e.g., a storage system subscriber) or requesting application may submita request for a data object with a given user key (without specifying aversion-id) through a web services interface (WS). The web serviceinterface may make a GETNEAREST call to keymap subsystem, which mayroute it to an appropriate KFC, as in 1620. For example, a hash routingcomponent of a keymap subsystem may determine an appropriate KFC towhich the request should be routed using a consistent hashing scheme, asdescribed above.

As illustrated in this example, if a latest symbolic key entry exists inthe KFC cache for this user key, shown as the positive exit from 1630,the method may include the KFC returning the keymap information for thelatest version to the web services interface (e.g., as identified in thelatest symbolic key entry), as in 1635. The method may also include theKFC adding or updating an entry with the keymap information for latestversion in its cache. If no latest symbolic key entry exists in the KFCcache for this user key, shown as the negative exit from 1630, themethod may include the KFC making a GETNEAREST call to the brick managerspecifying this user key, as in 1640. In some embodiments, the brickmanager may then determine the latest version-id for this user key, andmay return it (along with an indication that this is the latestversion-id and/or the keymap information associated with this latestversion-id) to the KFC, as in 1650. For example, the brick manager mayexamine two or more keymap information entries stored in persistentstorage to determine the latest version of the object (e.g., bycomparing a sequencer or timestamp portion of the version-id in each ofthe entries, by comparing the two leading entries in keymap informationthat is sorted by user key and then by version-id, as described above,or by any other suitable means).

In some embodiments, in response to receiving keymap information for thelatest version-id (and an indication of the latest version-id value),the KFC may add or update the values in two cache entries accordingly:the entry corresponding to the latest version-id, and the symboliclatest key entry for this user key, as in 1660. The KFC may then returnthe keymap information (including, for example, a inode) for the latestversion of the specified user key to the web services interface, as in1670. Note that the web services interface may use this information tolocate and fetch the desired data object itself in the distributedstorage system (not shown). For example, in some embodiments, the webservices interface may communicate some or all of the keymap informationto a storage node interface in the distributed storage unit (such asstorage node interface 640 in FIG. 6).

In some embodiments, keymap information (including, in some cases, alatest symbolic key entry) may need to be updated following an operationto delete a data object instance stored in the storage system. Oneembodiment of a method for updating keymap information following aDELETE operation in a distributed storage system is illustrated by theflow diagram in FIG. 17. Note that this method may or may not beapplicable in the case of a logical delete (described above), which maybe treated in the keymap subsystem as if it were a PUT operation for anew delete marker object. As illustrated at 1710, in this example, themethod may include a requester (e.g., a user or requesting application)initiating a DELETE type operation that specifies a user key. The webservices interface may send the delete request to the keymap subsystem,which may route the request to an appropriate KFC for processing, as in1720. As illustrated in this example, the method may include the KFCwriting a special sentinel value in the latest symbolic key entry forthe specified user key indicating that the “nearest” version-id valuefor this user key may change, as in 1730. Note that overwriting thelatest symbolic key entry may be used to effectively invalidate thelatest symbolic key entry, on the assumption that the DELETE will affectthe latest symbolic key entry. The method may also include the KFCadding (or updating) a cache entry for this request and setting thevalue in this cache entry to a special sentinel value (e.g., the samesentinel value or a different sentinel value, in various embodiments) toindicate that an operation is in progress.

As illustrated at 1740 in FIG. 17, the KFC may forward the deleterequest to an appropriate brick manager to update the keymap informationfor the object instance that is the target of the DELETE operationspecified in the request, as in 1750. In some embodiments, updating thekeymap information for the targeted object may include the brick managerdeleting the keymap information for the targeted data object instancefrom persistent storage. In other embodiments, e.g., embodiments inwhich delete markers are stored in the place of deleted object instancesfor a given user key rather than actually deleting those objectinstances, the brick manager may not delete the keymap information forthe targeted data object instance for the given user key, whether or notthe data object instance is itself deleted. In either case, the brickmanager may return a response to the KFC indicating that the DELETEoperation has been performed.

As illustrated at 1760 in this example, once the DELETE operation hasbeen performed, the method may include the KFC removing the latestsymbolic key entry for this user key (if there are no pending requestsinvolving this user key). The KFC may also update the cache entry forthis DELETE operation to replace the sentinel value with a specialtombstone value, as in 1770. This special tombstone value in the cacheentry may indicate that a DELETE operation has been performed on a dataobject (e.g., a particular data object instance) with the specified userkey. The KFC may return a response to the web services interfaceindicating that the DELETE operation has been performed, and the webservices interface may in turn return a similar response to therequester, as in 1780.

In some embodiments, rather than continuing to add entries to a cache ina keymap subsystem (e.g., latest symbolic key entries) withoutexplicitly replacing cache entries that are no longer needed, a storagesystem may include mechanisms for removing extraneous entries from thecache. One embodiment of a method for removing cache entries that are nolonger needed in a keymap subsystem is illustrated by the flow diagramin FIG. 18. As illustrated in this example, the method may include akeymap coordinator (KFC) receiving a request from a web servicesinterface (WS) to store keymap information for a data object instancebeing PUT, as in 1810, and the request may specify a user key. Forexample, a request to PUT a data object instance may be received by theweb services interface from a user or requesting application, indifferent embodiments. As illustrated at 1820 in this example, the KFCmay create or update a latest symbolic key entry for the specified userkey, and may write a special sentinel value in this cache entry. Themethod may also include the KFC adding (or updating) a cache entry forthis request and setting the value in this cache entry to a specialsentinel value (e.g., the same sentinel value or a different sentinelvalue, in various embodiments).

As illustrated in this example, the method may include the KFCincrementing an in-flight request tracker for the specified user key, asin 1830. The KFC may send the request to a brick manager (which maystore the keymap information for this PUT operation), as in 1840. Asillustrated in FIG. 18, the KFC may receive a response to the requestfrom the brick manager indicating that the keymap information has beenstored (and, in some cases, indicating whether the data object instanceis the latest version of the data object), or the request may time outwithout the brick manager sending a response to the KFC. After receivinga response (or after a pre-determined timeout period has expired), themethod may include the KFC decrementing the in-flight request trackerfor this user key, as in 1850. If the response from the brick managerindicates that the object being PUT is the latest version of the objectswith the specified user key, e.g., if the is-nearest parameter value isNEAREST (shown as the positive exit from 1860), the method may includethe KFC updating the latest symbolic key entry for this user key, i.e.replacing the sentinel value that was mapped to the latest symbolic keywith the version-id of the newly PUT object, as in 1865.

As illustrated in this example, if the response from the brick managerdoes not indicate that the object being PUT is the latest version of theobjects with the specified user key, e.g., if the is-nearest parametervalue is NOTNEAREST or UNKNOWN (shown as the negative exit from 1860),the KFC may query or examine the in-flight request tracker to determinewhether there are any pending requests for this user key. If thein-flight request tracker indicates that there are still pendingrequests (e.g., pending requests to store or retrieve keymapinformation) for this user key (shown as the positive exit from 1870),the method may include the KFC maintaining the latest symbolic key entryfor this user key (i.e. maintaining the sentinel value in the cacheentry for the latest symbolic key) in its cache, as in 1890. If thein-flight request tracker indicates that there are no pending requestsfor this user key (shown as the negative exit from 1870), the method mayinclude the KFC removing the latest symbolic key entry for this user key(i.e. the cache entry for the latest symbolic key containing the specialsentinel value), as in 1880. Note that in some embodiments, the latestsymbolic key entry may not be removed (e.g., at 1880) if anotheroperation has already caused the sentinel value to be replaced with theversion-id of another data object instance. In still other embodiments,the value of the latest symbolic key entry may not be maintained (e.g.,at 1890) if another operation has already caused the sentinel value tobe replaced with the version-id of another data object instance (e.g.,it may instead be removed).

As described above, in some embodiments, keymap information for storeddata objects may be cached as key-value pairs in a distributed hashtable, e.g, a distributed hash table in a keymap subsystem of adistributed storage system. A distributed hash table is a hash tablespread across many computing nodes (i.e. machines) within a fleet ofcomputing nodes. Each participant machine in the hash table may hold asubset of the data stored in the hash table. One potentially difficulttask when using a distributed hash table is finding a computing node inthe fleet that contains the data a requester (e.g., a user or requestingapplication) is looking for.

In consistent hashing, a hash function is executed in order to identifythe machine within the fleet that should house the data associated witha key (e.g., a user key or a composite key). This hash function issometimes referred to as the routing algorithm. In general, a hashfunction maps a key-value pair in the key domain to a numerical index inanother domain. For example, a routing algorithm may apply a hashfunction to the key of a key-value pair in order to transform the keyfrom its native domain type into the domain space of possible cachenodes within the distributed fleet. Once the transformation is applied,the caller has identified a machine that could contain the key withinthe fleet.

Given a good hash function and key space, consistent hashing may producea good distribution of keys across the fleet of machines. However, keyswhich as “close” to one another within their own domain spaces may notend up on the same machine in the fleet or even close to each otherwithin the distributed system. The result may be that keys that need tobe close to one another to support operations efficiently (such as a“GETNEAREST” operation in a keymap subsystem) may not be close to oneanother.

In some embodiments, the systems described herein extend the routingalgorithm described above so that a key may be bisected into two parts:a part of the key to which the hash function is applied in the routingalgorithm, and another part of the key to which the hash function is notapplied (and which does not affect the operation or results of therouting algorithm). In some embodiments, for example, when two or morekeys differ only in the latter part, they may both be stored on the samemachine. As described herein, this mechanism may in some embodiments beemployed in a storage system that supports versioning, and may allow thestorage system to leverage “user key” locality when executing variousAPIs, e.g., GETNEAREST.

One embodiment of a method for clustering keys in a distributed storagesystem is illustrated by the flow diagram in FIG. 19. As illustrated in1910, the method may include the storage system receiving a request tostore a data object in the distributed storage system, and the requestmay specify a particular composite key, which may include a sharableuser key, a version-id, and/or a locator identifying a particular objectinstance. As illustrated in this example, the method may include storingthe data object in the distributed storage system, as in 1920, andgenerating a keymap entry for the data object, as in 1930. In variousembodiments, the keymap entry may include the entire composite key,and/or any of the user key, a version-id (which may include a sequenceror other timestamp), and/or an object instance locator identifying aparticular instance or copy of a data object having the specified userkey. Note that some of this information (e.g., a user key portion of acomposite key) may be specified by a user (e.g., a storage servicesubscriber) or a requesting application, while other information may begenerated by the web services interface or another component of thestorage system in response to receiving a user request.

The method may include applying a hash function only to a portion of thecomposite key (i.e. to less than all of the bits of the key), as in1940. For example, in some embodiments, the composite key may includemultiple fields or encodings, and the hash function may be applied to orexclude one or more of the fields or encodings of the composite key. Insome embodiments, the hash function may be applied to or exclude apre-determined number of bits of the composite key. In variousembodiments, the hash function may be applied to or exclude the sameportions, fields, encodings, or bits for all composite keys or may beapplied to or exclude different portions, fields, encodings, or bits fordifferent composite keys. In one embodiment, a version-id delimiter of acomposite key (as described above) may be used to identify the portionof the key to which the hash function should be applied. In thisexample, only the user key portion of the composite key and not theversion-id (or sequencer or ID portion thereof) may be used by the hashrouter to determine the location at which the keymap information shouldbe stored, thus clustering keymap information for data object instanceshaving the same user key on a single computing node and/or in a cachethereof. Applying a hash function to other portions, partitions orsubsets of other types of composite keys or user keys (including lessthan all of the bits of a user key) may facilitate the clustering ofrelated user keys, of keys stored in same a time range, or keys havingany other common (or similar) element, in various embodiments.

As illustrated in this example, the method may include determining alocation in a distributed hash table at which to store the keymap entrydependent on the results of the hashing, as in 1950. For example,different ranges of hash values may be mapped to a respective one of aplurality of keymap coordinators. In various embodiments, each keymapcoordinator may cache the keymap information for subsequent use. Asillustrated in this example, and described above, by determining alocation in the distributed hash table at which to store the keymapentry using a hash value for a portion of the key (rather than theentire key), the keymap entry may be stored at a location in thedistributed hash table near one or more related keymap entries, as in1960.

One embodiment of a method for routing access requests for keymapinformation in a distributed storage system (e.g., requests to read orwrite keymap entries) is illustrated by the flow diagram in FIG. 20. Asillustrated in this example, the method may include generating a keymapentry for a data object stored in a distributed storage system thatincludes a composite key, and the composite key may include a sharableuser key, a version-id, and/or a locator identifying a particular objectinstance, as in 2010. Note that some of this information (e.g., a userkey portion of a composite key) may be specified by a user (e.g., astorage service subscriber) or a requesting application, while otherinformation may be generated by the web services interface or anothercomponent of the storage system in response to receiving a user request.As illustrated in FIG. 20 and described above, in some embodiments, themethod may include applying a hash function to a portion of thecomposite key, as in 2020, and routing the keymap entry to a particularmachine among multiple machines implementing a distributed hash tablebased on a mapping of hash value ranges to machines, as in 2030. Thekeymap entry may be stored on the particular machine mapped to the rangeof hash values that includes the hash value generated from a portion ofthe composite key, as in 2040, e.g., in a KFC cache on the particularmachine.

As illustrated at 2050 in FIG. 20, the storage system may receive arequest for keymap information for a data object (e.g., from a user orapplication) including at least a portion of a composite key, e.g., auser key, and (in some cases) a version-id (e.g., a request for therecently cached keymap information or for keymap information for anotherdata object instance). If the request does not specify a version-id thestorage system may determine the latest version of the data objecthaving the user key included in the request, as described herein. Asillustrated in this example, a method for routing requests to access(read) keymap information may include applying a hash function to aportion of the composite key (or elements thereof) included in therequest, as in 2060, and routing the keymap information request to agiven machine among machines implementing the distributed hash tablebased on a mapping of hash value ranges to machines, as in 2070. Forexample, the keymap information request may be routed to a machine thatis mapped to the range of hash values that includes the hash valuegenerated from a portion of the composite key (or elements thereof)included in the request. The given machine may then return the requestedkeymap information to the requester (e.g., the user or requestingapplication), as in 2080. Note that the operations illustrated in FIG.20 may be repeated for any number of requests for keymap information,e.g., for GET type operations, or GETNEAREST type operations.

One embodiment of a method for determining the latest version of anobject in a distributed storage system in which keymap information isstored on particular machines based on a consistent hashing of a portionof a key is illustrated by the flow diagram in FIG. 21. As illustratedin this example, the method may include a web server making a GETNEARESTcall to a keymap subsystem, as in 2110, and the call may specify acomposite key (which may include at least a user key). As describedabove, in some embodiments the keymap subsystem may apply a hashfunction to a portion of the composite key, as 2120, and the keymapsubsystem may route the GETNEAREST call to an appropriate keymapcoordinator (KFC), as in 2130. For example, a hash routing component ofthe keymap subsystem may route the call to a machine mapped to a rangeof hash values that includes the hash value generated from a portion ofthe specified composite key, and this machine may also cache keymapentries for other data object instances having the same user key as thatincluded in the request.

As illustrated in this example, if a valid latest symbolic key entry forthis user key exists in the KFC's cache, shown as the positive exit from2140, the keymap subsystem may return the keymap information stored inlatest symbolic key entry, as in 2145. If there is no valid latestsymbolic key entry for this key in the KFC's cache (e.g., if no suchentry exists, or if a latest symbolic key entry from this user keyincludes a special sentinel value rather than a valid version-id value),shown as the negative exit from 2140, the method may include the KFCmaking a GETNEAREST call to a brick manager specifying this user key, asin 2150, and the brick manager may determine the latest version of thestored objects having the specified user key. The brick manager may thenreturn the keymap information for the latest version of the objectshaving the specified key (and the version-id of the latest version) tothe KFC. Note that the KFC cache may be more likely to include a validlatest symbolic key entry for a given user in embodiments in which allkeymap entries for data object instances having the same user key aremapped to the same KFC.

Note that in embodiments in which keymap information is stored onparticular computing nodes and/or in particular storage partitions orblocks (bricks) based on a consistent hashing of a portion of a key, thebrick manager may only need to access one storage partition or block(brick) on one computing node to examine multiple keymap entries for akey (since this routing mechanism may result in the clustering of keymapentries for a given user key on the same storage partition or block(brick). In some embodiments, the leading keymap entry for the givenuser key, its neighbor, and/or a latest symbolic key entry for this userkey may all be present in the same KFC cache according to this routingmechanism.

As previously noted, various hash value ranges may be mapped to one ofthe keymap coordinators (KFCs) in a keymap subsystem of a distributedstorage system. In some embodiments, each keymap coordinator may beassociated with a single hash value range, while in other embodiments,some or all of the keymap coordinators in a keymap subsystem may beassociated with multiple ranges of hash values. This multiplicity inmapping may be further illustrated by the block diagram illustrated inFIG. 22. In this example, the set of potential, or known hash values,may be represented as a ring 2200. The hash values represented on ring2200 may be mapped to one of six keymap coordinators (identified in FIG.22 by the labels KFC1-KFC6), and the points on ring 2200 that arelabeled with these KFC identifiers indicate the last hash value that ismapped to the identified KFC.

As illustrated in FIG. 22, ring 2200 of potential hash values may bebroken down into a set of hash value ranges. In the example illustratedin FIG. 22, as in the example illustrated in FIGS. 9A and 9B, the hashvalue ranges defined for each keymap coordinator may be of differentlengths. In other embodiments, the hash value ranges defined for eachkeymap coordinator may be of equal size. In this example, the hash valueranges mapped to various KFCs include the hash value ranges labeled asregion R1 (between points 2206 and 2208 of ring 2200) and region R2(between points 2210 and 2212 of ring 2200), and these hash value rangesare mapped to different KFCs (e.g., KFC4 and KFC3, respectively). Inthis example, the hash value ranges labeled as region R2 (between points2210 and 2212 of ring 2200) and region R3 (between points 2222 and 2224of ring 2200) are mapped to the same KFC (KFC3). Again note that theassignment of ranges with ring 2000 may not necessarily follow aconsistent order or pattern. For example, in various embodiments, theorder of the assignment of hash value ranges to keymap coordinators maybe random, or may be determined in accordance with selection/processingcriteria other than, or in addition to, a consistent hashing scheme.

In some embodiments, the system and methods described herein for cachingand retrieving keymap information and for routing access requests forkeymap information may be employed by a storage service that providesstorage for subscribers as part of a virtualized computing service. Invarious embodiments, virtualized computing may be offered as anon-demand, paid service to clients, and may include a virtualizedstorage service, which may in some embodiments support objectversioning, as described herein. For example, an enterprise may assembleand maintain the various hardware and software components used toimplement virtualized computing, and may offer clients access to theseresources according to various pricing models (e.g., usage-basedpricing, subscription pricing, etc.). Thus, clients may have access to arange of virtual computing resources without having to incur the costsof provisioning and maintaining the infrastructure needed to implementthose resources.

Example Computer System Embodiment

It is contemplated that in some embodiments, any of the methods,techniques or components described herein may be implemented asinstructions and data capable of being stored or conveyed via acomputer-accessible medium. Such methods or techniques may include, forexample and without limitation, various methods of caching andretrieving keymap information and for routing access requests for keymapinformation, as described herein. Such instructions may be executed toperform specific computational functions tailored to specific purposes(e.g., processing requests received via a web services interface;storing, retrieving, modifying and/or otherwise accessing data objectsand/or access control lists and metadata thereof; maintaining multipleversions of stored data objects; caching and retrieving keymapinformation; and routing access requests for keymap information, asdescribed herein) as well as higher-order functions such as operatingsystem functionality, virtualization functionality, networkcommunications functionality, application functionality, storage systemfunctionality, and/or any other suitable functions.

One example embodiment of a computer system that includescomputer-accessible media and that supports caching and retrievingkeymap information and routing access requests for keymap informationusing the mechanisms described herein is illustrated in FIG. 23. Invarious embodiments, the functionality of any of the various modules ormethods described herein may be implemented by one or several instancesof computer system 2300. In particular, it is noted that differentelements of the system described herein may be implemented by differentcomputer systems 2300. For example, a storage system that supports thefunctionality described herein for caching and retrieving keymapinformation and for routing keymap information requests may beimplemented on the same computer system 2300 on which a client (throughwhich a user/requester accesses the storage system) executes, or onanother computer system 2300, in different embodiments. In anotherexample, data objects may be stored on one or more of a plurality ofcomputing nodes in a distributed storage system, a keymap subsystem maybe implemented on one or more of the computing nodes, and keymapinformation may stored in a distributed hash table across multiple onesof the computing nodes, and each of the computing nodes may be similarto computer system 2300.

In the illustrated embodiment, computer system 2300 includes one or moreprocessors 2310 coupled to a system memory 2320 via an input/output(I/O) interface 2330. Computer system 2300 further includes a networkinterface 2340 coupled to I/O interface 2330. In various embodiments,computer system 2300 may be a uniprocessor system including oneprocessor 2310, or a multiprocessor system including several processors2310 (e.g., two, four, eight, or another suitable number). Processors2310 may be any suitable processor capable of executing instructions.For example, in various embodiments processors 2310 may be ageneral-purpose or embedded processor implementing any of a variety ofinstruction set architectures (ISAs), such as the x86, PowerPC™, SPARC™,or MIPS™ ISAs, or any other suitable ISA. In multiprocessor systems,each of processors 2310 may commonly, but not necessarily, implement thesame ISA.

System memory 2320 may be configured to store instructions (e.g., code2325) and data (e.g., in data store 2322) accessible by processor 2310.In various embodiments, system memory 2320 may be implemented using anysuitable memory technology, such as static random access memory (SRAM),synchronous dynamic RAM (SDRAM), nonvolatile/Flash-type memory, or anyother type of memory. In the illustrated embodiment, instructions anddata implementing desired functions, methods or techniques (such asfunctionality for supporting versioning of stored data objects, forperforming various operations to store, retrieve, modify and otherwiseaccess data objects and/or access control lists thereof on a storagesystem, for caching keymap information, and for routing access requestsfor keymap information in the storage system according to the APIs andother mechanisms described herein), are shown stored within systemmemory 2320 as code 2325. It is noted that in some embodiments, code2325 may include instructions and data implementing desired functionsthat are not directly executable by processor 2310 but are representedor encoded in an abstract form that is translatable to instructions thatare directly executable by processor 2310. For example, code 2325 mayinclude instructions specified in an ISA that may be emulated byprocessor 2310, or by other code 2325 executable on processor 2310.Alternatively, code 2325 may include instructions, procedures orstatements implemented in an abstract programming language that may becompiled or interpreted in the course of execution. As non-limitingexamples, code 2325 may include code specified in a procedural orobject-oriented programming language such as C or C++, a scriptinglanguage such as perl, a markup language such as HTML or XML, or anyother suitable language.

In some embodiments, objects (e.g., data objects and/or delete markerobjects in one or more buckets) and/or access control lists and othermetadata thereof may be stored in a data store 2322 within system memory2320. In some embodiments, data store 2322 may store one or more tablesindicating the mapping of hash value ranges to computing nodes orpartitions of a distributed hash table. In some embodiments, systemmemory 2320 may include persistent storage in which data objects and/orkeymap information are stored in the distributed storage system. In someembodiments, computer system 2300 may be a computer system on which akeymap subsystem of a distributed storage system is implemented, andsystem memory 2320 may include one or more caches 2324, such as thecaches associated with each keymap coordinator (KFC) in the keymapsubsystem.

In one embodiment, I/O interface 2330 may be configured to coordinateI/O traffic between processor 2310, system memory 2320, and anyperipheral devices in the device, including network interface 2340 orother peripheral interfaces. In some embodiments, I/O interface 2330 mayperform any necessary protocol, timing or other data transformations toconvert data signals from one component (e.g., system memory 2320) intoa format suitable for use by another component (e.g., processor 2310).In some embodiments, I/O interface 2330 may include support for devicesattached through various types of peripheral buses, such as a variant ofthe Peripheral Component Interconnect (PCI) bus standard or theUniversal Serial Bus (USB) standard, for example. In some embodiments,the function of I/O interface 2330 may be split into two or moreseparate components, such as a north bridge and a south bridge, forexample. Also, in some embodiments some or all of the functionality ofI/O interface 2330, such as an interface to system memory 2320, may beincorporated directly into processor 2310.

Network interface 2340 may be configured to allow data to be exchangedbetween computer system 2300 and other devices attached to a network,such as other computer systems, for example. In various embodiments,network interface 2340 may support communication via wired or wirelessgeneral data networks, such as any suitable type of Ethernet network,for example; via telecommunications/telephony networks such as analogvoice networks or digital fiber communications networks; via storagearea networks such as Fibre Channel SANs, or via any other suitable typeof network and/or protocol.

In some embodiments, system memory 2320 may include a non-transitory,computer-accessible storage medium configured to store instructions anddata as described above. However, in other embodiments, instructionsand/or data may be received, sent or stored upon different types ofcomputer-accessible storage media. Generally speaking, acomputer-accessible storage medium may include storage media or memorymedia such as magnetic or optical media, e.g., disk or CD/DVD-ROMcoupled to computer system 2300 via I/O interface 2330. Acomputer-accessible storage medium may also include any volatile ornon-volatile storage media such as RAM (e.g. SDRAM, DDR SDRAM, RDRAM,SRAM, etc.), ROM, etc, that may be included in some embodiments ofcomputer system 2300 as system memory 2320 or another type of memory. Acomputer-accessible storage medium may generally be accessible viatransmission media or signals such as electrical, electromagnetic, ordigital signals, conveyed via a communication medium such as a networkand/or a wireless link, such as may be implemented via network interface2340.

Although the embodiments above have been described in considerabledetail, numerous variations and modifications will become apparent tothose skilled in the art once the above disclosure is fully appreciated.It is intended that the following claims be interpreted to embrace allsuch variations and modifications.

What is claimed is:
 1. A method, comprising: performing, by a computersystem that stores a plurality of data objects in a distributed storagesystem: storing an instance of a data object, wherein storing theinstance of the data object comprises: receiving, from an entity otherthan the distributed storage system, a request to store the instance ofthe data object in the distributed storage system, wherein the requestcomprises a user key for the data object and a version identifier forthe instance of the data object; generating keymap information for theinstance of the data object that maps the user key to a locator and thelocator to the instance of the data object; caching the keymapinformation at a keymap coordinator in the distributed storage system;caching a latest symbolic key entry at the keymap coordinator, whereinthe latest symbolic key entry comprises the version identifier for theinstance of the data object and indicates that the instance of the dataobject is the latest version of the data object stored in thedistributed storage system; providing, responsive to a request toretrieve the latest version of the data object, the latest version ofthe data object, wherein the request does not specify the version of thedata object to be obtained, wherein providing the latest version of thedata object comprises: receiving, from an entity other than thedistributed storage system, the request to retrieve the latest versionof the data object, wherein the request comprises the user key for thedata object but does not include a version identifier of the latestversion of the data object; determining the version identifier of thelatest version of the data object dependent on the cached latestsymbolic key entry; and returning, based on the determined versionidentifier, the latest version of the data object having the user key.2. The method of claim 1, wherein said determining the versionidentifier of the latest version of the data object comprises obtainingthe version identifier from the latest symbolic key entry cached at thekeymap coordinator.
 3. The method of claim 1, further comprising: priorto said caching the latest symbolic key entry at the keymap coordinator,determining that the instance of the data object is the latest versionof the data object stored in the distributed storage system.
 4. Themethod of claim 3, further comprising: setting a flag to indicate thatthe latest symbolic key entry should be updated; wherein saiddetermining that the instance of the data object is the latest versionof the data object is performed in response to the flag being set. 5.The method of claim 1, further comprising, prior to said receiving arequest to retrieve the latest version of the data object: receiving arequest to store another instance of the data object in the distributedstorage system, wherein the request comprises the user key for the dataobject and a version identifier for the other instance of the dataobject; generating keymap information for the other instance of the dataobject; caching the keymap information for the other instance of thedata object at the keymap coordinator; and updating the cached latestsymbolic key entry at the keymap coordinator, wherein the updated latestsymbolic key entry comprises the version identifier for the otherinstance of the data object and indicates that the other instance of thedata object is the latest version of the data object stored in thedistributed storage system; wherein said determining the versionidentifier of the latest version of the data object comprises obtainingthe version identifier from the updated latest symbolic key entry cachedat the keymap coordinator.
 6. A method, comprising: performing, by acomputer system that stores a plurality of data objects in a distributedstorage system: caching keymap entries for one or more data objectinstances stored in the distributed storage system, wherein each keymapentry maps a user key to a locator and the locator to an instance of adata object having the user key, wherein multiple versions of theinstance of the data object have the same user key; caching a latestsymbolic key entry for at least one of the user keys, wherein eachlatest symbolic key entry includes a version identifier for the latestversion of the data objects stored in the distributed storage systemhaving a particular user key; receiving a request to retrieve the latestversion of a data object, wherein the request specifies a user key forthe data object but does not include a version identifier of the latestversion of the data object; determining the version identifier of thelatest version of the data object dependent on whether a latest symbolickey entry for the specified user key is cached in the distributedstorage system; and returning keymap information for the latest versionof the data object.
 7. The method of claim 6, wherein a latest symbolickey entry for the specified user key is cached in the distributedstorage system; and wherein said determining the version identifier ofthe latest version of the data object comprises obtaining the versionidentifier from the latest symbolic key entry.
 8. The method of claim 6,further comprising storing the keymap entries for one or more dataobject instances in persistent storage in the distributed storagesystem.
 9. The method of claim 6, wherein no latest symbolic key entryfor the specified user key is cached in the distributed storage system;and wherein said determining the version identifier of the latestversion of the data object comprises examining two or more keymapentries stored in persistent storage in the distributed storage system.10. The method of claim 6, further comprising: receiving a request tostore keymap information for an instance of a data object having a newuser key; storing the keymap information for the instance of the dataobject having the new user key in persistent storage in the distributedstorage system; and creating a latest symbolic key entry for the newuser key, wherein the latest symbolic key entry for the new user keycomprises a version identifier for the instance of the data objecthaving the new user key and indicates that the instance of the dataobject having the new user key is the latest version of the data objecthaving the new user key stored in the distributed storage system. 11.The method of claim 6, receiving a request to store keymap informationfor an instance of a data object having an existing user key;determining whether it is possible that the instance of the data objecthaving the existing user key is the latest version of the data objectsstored in the distributed storage system having the existing user key;and in response to determining that it is possible that the instance ofthe data object having the existing user key is the latest version ofthe data objects stored in the distributed storage system having theexisting user key, setting a flag to indicate that a calculation of thelatest version of the data objects stored in the distributed storagesystem having the existing user key should be attempted.
 12. The methodof claim 11, wherein attempting to calculate the latest version of thedata objects stored in the distributed storage system having theexisting user key comprises: storing a special sentinel value in alatest symbolic key entry for the existing user key; examining one ormore keymap entries stored in persistent storage; and returning aresponse to the attempt.
 13. The method of claim 12, wherein theresponse includes an indication that the instance of the data objecthaving the existing user key is the latest version of the data objectsstored in the distributed storage system having the existing user key;and wherein the method further comprises: replacing the special sentinelvalue stored in the latest symbolic key entry for the existing user keywith a value indicating the version identifier of the instance of thedata object having the existing user key.
 14. The method of claim 12,wherein the response includes an indication that the instance of thedata object having the existing user key is not the latest version ofthe data objects stored in the distributed storage system having theexisting user key or that it is unknown whether the instance of the dataobject having the existing user key is the latest version of the dataobjects stored in the distributed storage system having the existinguser key; and wherein the method further comprises: removing the latestsymbolic key entry for the existing user key from the cache.
 15. Themethod of claim 14, wherein said removing the latest symbolic key entryfor the existing user key from the cache is performed in response todetermining that there are no pending requests to access keymapinformation for data objects having the existing user key.
 16. Themethod of claim 11, further comprising prior to determining that it ispossible that the instance of the data object having the existing userkey is the latest version of the data objects stored in the distributedstorage system having the existing user key, creating a cache entry forthe instance of the data object having the existing user key, andstoring a special sentinel value in the cache entry; and in response todetermining that the instance of the data object having the existinguser key is the latest version of the data objects stored in thedistributed storage system having the existing user key, replacing thespecial sentinel value stored in the cache entry for the instance of thedata object with keymap information for the instance of the data object.17. The method of claim 6, wherein said caching keymap entries comprisesstoring the keymap entries on two or more computing nodes of thecomputer system in a distributed hash table, and wherein each keymapentry is indexed in the distributed hash table by hash value generatedfrom at least a portion of a composite key that includes the user keyand that uniquely identifies a data object instance stored in thedistributed storage system.
 18. The method of claim 17, wherein keymapentries for two or more data object instances having the same user keyare stored on the same computing node in the distributed hash table. 19.A non-transitory, computer-readable storage medium storing programinstructions that when executed on one or more computers cause the oneor more computers to perform: caching keymap entries for one or moredata object instances stored in a distributed storage system, whereineach keymap entry maps a user key to a locator and the locator to aninstance of a data object having the user key, wherein multiple versionsof the instance of the data object have the same user key; caching alatest symbolic key entry for at least one of the user keys, whereineach latest symbolic key entry includes a version identifier for thelatest version of the data objects stored in the distributed storagesystem having a particular user key; receiving a request to retrieve thelatest version of a data object, wherein the request specifies a userkey for the data object but does not include a version identifier of thelatest version of the data object; determining the version identifier ofthe latest version of the data object dependent on whether a latestsymbolic key entry for the specified user key is cached in thedistributed storage system; and returning keymap information for thelatest version of the data object.
 20. The storage medium of claim 19,wherein a latest symbolic key entry for the specified user key is cachedin the distributed storage system; and wherein said determining theversion identifier of the latest version of the data object comprisesobtaining the version identifier from the latest symbolic key entry. 21.The storage medium of claim 19, wherein when executed on the one or morecomputers, the program instructions further cause the one or morecomputers to perform storing the keymap entries for one or more dataobject instances in persistent storage in the distributed storagesystem; wherein no latest symbolic key entry for the specified user keyis cached in the distributed storage system; and wherein saiddetermining the version identifier of the latest version of the dataobject comprises examining two or more keymap entries stored inpersistent storage in the distributed storage system.
 22. The storagemedium of claim 19, wherein when executed on the one or more computers,the program instructions further cause the one or more computers toperform: receiving a request to store keymap information for an instanceof a data object having a new user key; storing the keymap informationfor the instance of the data object having the new user key inpersistent storage in the distributed storage system; and creating alatest symbolic key entry for the new user key, wherein the latestsymbolic key entry for the new user key comprises a version identifierfor the instance of the data object having the new user key andindicates that the instance of the data object having the new user keyis the latest version of the data object having the new user key storedin the distributed storage system.
 23. The storage medium of claim 19,wherein when executed on the one or more computers, the programinstructions further cause the one or more computers to perform:receiving a request to store keymap information for an instance of adata object having an existing user key; determining whether it ispossible that the instance of the data object having the existing userkey is the latest version of the data objects stored in the distributedstorage system having the existing user key; and in response todetermining that it is possible that the instance of the data objecthaving the existing user key is the latest version of the data objectsstored in the distributed storage system having the existing user key,setting a flag to indicate that a calculation of the latest version ofthe data objects stored in the distributed storage system having theexisting user key should be attempted.
 24. The storage medium of claim23, wherein attempting to calculate the latest version of the dataobjects stored in the distributed storage system having the existinguser key comprises: storing a special sentinel value in a latestsymbolic key entry for the existing user key; examining one or morekeymap entries stored in persistent storage; and returning a response tothe attempt.
 25. The storage medium of claim 24, wherein the responseincludes an indication that the instance of the data object having theexisting user key is the latest version of the data objects stored inthe distributed storage system having the existing user key; and whereinwhen executed on the one or more computers, the program instructionsfurther cause the one or more computers to perform: replacing thespecial sentinel value stored in the latest symbolic key entry for theexisting user key with a value indicating the version identifier of theinstance of the data object having the existing user key.
 26. Thestorage medium of claim 24, wherein the response includes an indicationthat the instance of the data object having the existing user key is notthe latest version of the data objects stored in the distributed storagesystem having the existing user key or that it is unknown whether theinstance of the data object having the existing user key is the latestversion of the data objects stored in the distributed storage systemhaving the existing user key; and wherein when executed on the one ormore computers, the program instructions further cause the one or morecomputers to perform: removing the latest symbolic key entry for theexisting user key from the cache.
 27. The storage medium of claim 26,wherein said removing the latest symbolic key entry for the existinguser key from the cache is performed in response to determining thatthere are no pending requests to access keymap information for dataobjects having the existing user key.
 28. The storage medium of claim23, wherein when executed on the one or more computers, the programinstructions further cause the one or more computers to perform: priorto determining that it is possible that the instance of the data objecthaving the existing user key is the latest version of the data objectsstored in the distributed storage system having the existing user key,creating a cache entry for the instance of the data object having theexisting user key, and storing a special sentinel value in the cacheentry; and in response to determining that the instance of the dataobject having the existing user key is the latest version of the dataobjects stored in the distributed storage system having the existinguser key, replacing the special sentinel value stored in the cache entryfor the instance of the data object with keymap information for theinstance of the data object.
 29. A system, comprising: a persistent datastore that stores a plurality of data objects, wherein each of theplurality of data objects comprises a user key and a version identifier;a cache; one or more processors; and a memory coupled to the one or moreprocessors and storing program instructions that when executed by theone or more processors cause the one or more processors to perform:receiving a request to store keymap information for an instance of adata object to be stored in the persistent data store, wherein therequest comprises a user key for the data object and a versionidentifier for the instance of the data object, and wherein the keymapinformation maps the user key to a locator and the locator to theinstance of the data object, wherein multiple versions of the instanceof the data object have the same user kev; storing the keymapinformation in an entry in the cache for the instance of the dataobject; storing a latest symbolic key entry in the cache, wherein thelatest symbolic key entry comprises the version identifier for theinstance of the data object and indicates that the instance of the dataobject is the latest version of the data object stored in thedistributed storage system; receiving a request to retrieve the latestversion of the data object, wherein the request comprises the user keyfor the data object but does not include a version identifier of thelatest version of the data object; determining the version identifier ofthe latest version of the data object dependent on the cached latestsymbolic key entry; and returning keymap information for the latestversion of the data object.
 30. The system of claim 29, wherein saiddetermining the version identifier of the latest version of the dataobject comprises obtaining the version identifier from the latestsymbolic key entry stored in the cache.
 31. The system of claim 29,wherein when executed by the one or more processors, the programinstructions further cause the one or more processors to perform: priorto said caching the latest symbolic key entry at the keymap coordinator,determining that the instance of the data object is the latest versionof the data object stored in the distributed storage system.
 32. Thesystem of claim 31, wherein when executed by the one or more processors,the program instructions further cause the one or more processors toperform: setting a flag to indicate that the latest symbolic key entryshould be updated; wherein said determining that the instance of thedata object is the latest version of the data object is performed inresponse to the flag being set.
 33. The system of claim 29, wherein whenexecuted by the one or more processors, the program instructions furthercause the one or more processors to perform, prior to said receiving arequest to retrieve the latest version of the data object: receiving arequest to store keymap information for another instance of the dataobject to be stored in the persistent data store, wherein the requestcomprises the user key for the data object and a version identifier forthe instance of the data object, and wherein the keymap information mapsthe user key to a locator and the locator to the instance of the dataobject; storing the keymap information for the other instance of thedata object in an entry in the cache; and updating the latest symbolickey entry in the cache, wherein the updated latest symbolic key entrycomprises the version identifier for the other instance of the dataobject and indicates that the other instance of the data object is thelatest version of the data object stored in the distributed storagesystem; wherein said determining the version identifier of the latestversion of the data object comprises obtaining the version identifierfrom the updated latest symbolic key entry stored in the cache.